Information processing method, program, and information processing device

Abstract

An object of the present invention is to provide an information processing method, a program, and an information processing device that can improve the quality of a composite image. [Solution] An information processing method for generating a composite image by synthesizing hand-drawn images is provided, the method comprising an image acquisition step, an image conversion step, and an image synthesis step, wherein the image acquisition step acquires a first image and a second image, the first image being an image represented in cube map projection and the second image being an image represented in equirectangular projection, the image conversion step converts the first image from an arrangement in the cube map projection to an arrangement in the equirectangular projection to generate a transformed first image, and the image synthesis step synthesizes at least the transformed first image and the second image to generate the composite image.

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing device. [Background technology]

[0002] Patent Document 1 discloses a technique for generating a panoramic image as a composite image by combining images captured by a digital camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-369060 Summary of the Invention [Problem to be solved by the invention]

[0004] When generating a composite image using multiple images, the quality of the composite image may be reduced if the process does not take into account the drawing methods of the multiple images. For example, when combining rendered images obtained using an image drawing application, if a drawing method that is not suitable for the image drawing application is used, it becomes difficult to draw the objects, resulting in a reduced quality of the resulting rendered image, and as a result, there is a concern that the quality of the composite image obtained by combining the rendered images may also be reduced.

[0005] An object of the present invention is to provide an information processing method, a program, and an information processing device that can improve the quality of a composite image. [Means for solving the problem]

[0006] According to the present invention, there is provided an information processing method for generating a composite image by synthesizing hand-drawn images, the information processing method comprising an image acquisition step, an image conversion step, and an image synthesis step, wherein the image acquisition step acquires a first image and a second image, the first image being an image represented in cube map projection and the second image being an image represented in equirectangular projection, the image conversion step converts the first image from an arrangement in the cube map projection to an arrangement in the equirectangular projection to generate a transformed first image, and the image synthesis step synthesizes at least the transformed first image and the second image to generate the composite image.

[0007] According to the present invention, an image suitable for representation using cube map projection can be used as the first image, and an image suitable for representation using equirectangular projection can be used as the second image, making it possible to select the projection method appropriate for the content of each image, thereby improving the quality of the composite image. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration example of an information processing system 100 that executes an information processing method according to an embodiment. [Figure 2] Fig. 2A shows an example of functional blocks of the information processing server 1 of the information processing system 100 shown in Fig. 1. Fig. 2B shows an example of functional blocks of the user terminal 3 of the information processing system 100 shown in Fig. 1. [Figure 3] FIG. 3 shows an example of functional blocks of the control unit 32 of the user terminal 3 shown in FIG. 2B. [Figure 4] Fig. 4A shows an example of each rendering region (region Rg11 to region Rg16) in cube map projection, and Fig. 4B shows an example of each rendering region (region Rg21 to region Rg26) in equirectangular projection. [Figure 5]Fig. 5 shows an example of the first image D1. In Fig. 5, the first image D1 is an image obtained by depicting pictures in each depiction region (region Rg11 to region Rg16) in the cube map projection shown in Fig. 4A. [Figure 6] 6A to 6F are enlarged views of the depiction areas (area Rg11 to area Rg16) in which pictures are depicted in FIG. [Figure 7] FIG. 7 shows a converted first image D11 obtained by performing image conversion processing (processing to convert an image from cube map projection to equirectangular projection) on the first image D1 shown in FIG. [Figure 8] Fig. 8 shows an example of the second image D2. In Fig. 8, the second image D2 is an image obtained by depicting pictures in each depiction area (area Rg21 to area Rg26) in the equirectangular projection shown in Fig. 4B. [Figure 9] 9A to 9C show an example of the third image. [Figure 10] FIG. 10 is an example of a flowchart relating to the information processing method of the embodiment. [Figure 11] FIG. 11A is an explanatory diagram that shows a schematic diagram of how a VR space is synthesized from an image in equirectangular projection, and FIG. 11B is an explanatory diagram that shows a schematic diagram of how a VR space is synthesized from an image in cube map projection. [Figure 12] 12A and 12B show images created using cube map projection that have been subjected to image conversion processing (processing to convert the image from cube map projection to equirectangular projection) and then converted into an image in VR space. FIG. 12A is an example of an image from above in VR space (an image viewed from above from a fixed position). FIG. 12B is an example of an image from below in VR space (an image viewed from below from a fixed position). [Figure 13] FIG. 13 shows an example of functional blocks of the control unit 32 of the user terminal 3 according to the modified example. [Figure 14] FIG. 14 is an example of a flowchart relating to an information processing method of the modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other. Furthermore, each feature can be an invention independently.

[0010] 1. System Configuration of Information Processing System 100 As shown in FIG. 1, the information processing system 100 includes an information processing server 1 configured to be connectable to a communication network 2 (e.g., the Internet, etc.), and a user terminal 3. The information processing server 1 is configured to be able to provide a program (image processing application) related to an information processing method according to the embodiment to the user terminal 3 (an example of an information processing device) via the communication network 2. The information processing server 1 and the user terminal 3 have functions such as data acquisition, generation, editing, storage, and output, and are configured to be able to process various types of data. The user terminal 3 may be, for example, a tablet terminal, a desktop PC, a laptop PC, a smartphone, or the like. In one example of the embodiment, the user terminal 3 is a tablet terminal capable of accepting pen input.

[0011] A user can generate a composite image by executing an image processing application on the user terminal 3 that the user owns. Here, the image processing application according to the embodiment has a function of generating a composite image from a hand-drawn picture image. In the embodiment, the composite image is an image obtained by combining a plurality of hand-drawn picture images (a converted first image D11 and a converted second image D2, which will be described later). The hand-drawn picture image can be generated, for example, by a user operating an electronic pen and having the user terminal 3 (tablet terminal) accept the operation input. However, this is not limited to this, and an image of a hand-drawn picture drawn on a terminal other than the user terminal 3 may be imported into the user terminal 3, and a composite image may be generated from the imported image. Furthermore, the composite image according to the embodiment is a spherical image, and is an image that can be displayed in, for example, a VR space. A spherical image is an image with a 360-degree field of view, and can be displayed, for example, on a VR device or a display. When displayed on a display, for example, a user can view an image in any position and in any direction by manipulating the viewpoint using, for example, desired viewer software or a web service. Note that the composite image according to the embodiment may be a planar image displayed in a predetermined projection (for example, equirectangular projection) before being converted into a spherical image. In other words, the composite image according to the embodiment may not necessarily be a spherical image (an image in a VR space) as long as it has undergone processing to combine multiple images.

[0012] When executing an image processing application, the user terminal 3 may download the image processing application and communicate with the information processing server 1 to receive the service (native application method). Alternatively, the user terminal 3 may communicate with the information processing server 1 via a web browser to receive the service (web application method). Furthermore, communication between the user terminal 3 and the information processing server 1 may occur only when the image processing application is downloaded to the user terminal 3, and after the download is complete, the user terminal 3 may operate based on the image processing application program stored in the user terminal 3 without communicating with the information processing server 1 (a so-called standalone form).

[0013] Each component of the information processing system 100, such as the information processing server 1, has one or more functions (functional units). Each component may be configured as a single device as shown in Fig. 1, or may be configured as multiple independent devices configured to be able to exchange information. The same applies to each functional unit, such as the control unit 12 of the information processing server 1 and the control unit 32 of the user terminal 3, which will be described later.

[0014] 1-1. Information Processing Server 1 The configurations of the information processing server 1 and the user terminal 3 will be further described with reference to FIGS. 2A and 2B. In an example of the embodiment, the information processing server 1 and the user terminal 3 include functional units having similar functions.

[0015] That is, the information processing server 1 has a communication unit 10, a memory unit 11, a control unit 12, an output unit 13, and an input unit 14, and these components are electrically connected within the information processing server 1 via a communication bus 15. The user terminal 3 has a communication unit 30, a memory unit 31, a control unit 32, an output unit 33, and an input unit 34, and these components are electrically connected within the user terminal 3 via a communication bus 35.

[0016] Each of the above components may be implemented by software or hardware. When implemented by software, various functions can be realized by a CPU executing a computer program. The program may be stored on a non-transitory computer-readable recording medium, provided as a downloadable file from an external server, or implemented by so-called cloud computing, in which a program stored in an external storage unit is read and functions are realized. When implemented by hardware, it can be implemented by various circuits, such as an ASIC, FPGA, or DRP. The embodiments deal with various information and concepts encompassing such information. These are represented by high and low signal values ​​or quantum bits as a binary bit set consisting of 0 or 1, and communication and calculations can be performed by the above software or hardware aspects. The software may be a general-purpose OS or a dedicated OS.

[0017] In this section, since the functional units of the information processing server 1 and the user terminal 3 are similar, they will be simply referred to as information processing devices. The communication unit 10 and the communication unit 30 will be simply referred to as communication units, the memory unit 11 and the memory unit 31 will be simply referred to as memory units, the control unit 12 and the control unit 32 will be simply referred to as control units, the output unit 13 and the output unit 33 will be simply referred to as output units, and the input unit 14 and the input unit 34 will be simply referred to as input units.

[0018] The communication unit may employ a wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc. The communication unit may also employ a configuration in which it is connected to the communication network 2 via a wireless communication means such as wireless LAN network communication, mobile communication such as 3G / LTE / 5G, or Bluetooth (registered trademark) communication. The communication unit 10 may also be configured to use both the wired communication means and wireless communication means described above.

[0019] The memory unit stores, for example, various programs executed by the control unit, and various values ​​such as constants, variables, and setting values. The memory unit can also store, for example, data acquired by communicating with an external device or an external server. The memory unit can employ, for example, a storage device such as a solid-state drive (SSD), or a storage medium such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to program calculations. In addition to the memory unit, an external memory unit (for example, an external storage medium, cloud, etc.) may also be used.

[0020] The control unit is configured to execute an information processing method (information processing) according to an embodiment. The control unit can be configured, for example, by a central processing unit (CPU), and in the embodiment, the control unit is an example of a processor capable of executing a program according to the information processing method having the operations (steps) described below. The control unit realizes various functions, for example, by reading out a program stored in the storage unit. Furthermore, software information processing is realized, for example, by processing various programs stored in the storage unit by the control unit as hardware.

[0021] The output unit is, for example, a display unit of the information processing device. The output unit may be, for example, included in the housing of the information processing device or may be externally attached. The output unit displays a graphical user interface (GUI) screen that can be operated by a user. The output unit may be, for example, a display device such as a CRT display, a liquid crystal display, an organic EL display, a plasma display, or an electronic paper display, as well as a display device such as an illuminable light or a projector. It is optional whether or not the information processing device itself has an output unit. For example, the output of the information processing device may be displayed on a display separate from the information processing device, or the output of the information processing device may be displayed on a display unit located at a location independent of the location where the information processing device is installed.

[0022] The input unit is configured to receive, for example, an operation input made by a user of the information processing device. The input unit may be included in the housing of the information processing device or may be externally attached. For example, an electronic pen, a touch panel, a switch button, a mouse, a keyboard camera, a scanner, etc. may be used as the input unit. It is optional whether or not the information processing device itself includes an input unit. For example, the operation input to the information processing device may be received via a controller connected to the information processing device by wire or wirelessly, or the operation input to the information processing device may be received via another information processing device located at a location independent of the location where the information processing device is installed.

[0023] 2. Functional configuration The functional configuration of the control unit 32 according to this embodiment will be described with reference to Fig. 3 etc. For example, information processing by software stored in the storage unit 31 is specifically realized by the control unit 32, which is an example of hardware, and each functional unit included in the control unit 32 is executed.

[0024] 2-1. Depiction area display section 320 The drawing area display unit 320 has a function of displaying a drawing area in which a user can draw a handwritten picture using an electronic pen, as shown in Figures 4A and 4B. Figure 4A shows a first drawing area Rg1 that is displayed when a handwritten picture is drawn using cube map projection. Figure 4B shows a second drawing area Rg2 that is displayed when a handwritten picture is drawn using equirectangular projection.

[0025] 2-2. Image acquisition unit 321 The image acquisition unit 321 has a function of acquiring data of handwritten pictures input in the first drawing area Rg1 and the second drawing area Rg2 displayed on the display of the user terminal 3, which is a tablet terminal. When the user terminal 3 imports an image of a handwritten picture drawn on a terminal (information processing device) other than the user terminal 3, the image acquisition unit 321 has a function of importing the image of the handwritten picture. The functions of the image acquisition unit 321 may include not only functions related to sending and receiving data between hardware devices, but also processing for modifying data. Specifically, the functions of the image acquisition unit 321 may include, for example, preprocessing such as removing noise from the data.

[0026] 2-3. Image conversion unit 322 The image conversion unit 322 has a function of converting an image shown in the first projection (see the first image D1 in FIG. 5) into an image shown in the second projection (see the converted first image D11 in FIG. 7). In the embodiment, the first projection is a cube map projection or a planar projection, and the second projection is an equirectangular projection. Note that the first projection and the second projection are not limited to these, and various other projections can be used.

[0027] 2-4. Image synthesis unit 323 The image synthesis unit 323 has a function of synthesizing an image converted by the image conversion unit 322 (see converted first image D11 in FIG. 7) with a different hand-drawn image (second image D2 in FIG. 8). In the embodiment, the image synthesis unit 323 has a function of synthesizing not only these images but also images (third images) such as those shown in FIGS. 9A to 9C. It is sufficient to synthesize the converted first image D11 and the second image D2, and it is optional whether or not to synthesize this third image as well. 9A to 9C (third images) are hand-drawn pictures, and in the example embodiment, they show a girl dancing ballet. The images (third images) shown in Fig. 9A to 9C are switched sequentially, and the composite images become a composite video.

[0028] 3. Explanation of how we process information The information processing method according to the embodiment will be described with reference to the flowchart of FIG. In the following description, the order of processes (steps) can be changed as appropriate, multiple processes can be executed simultaneously, or some processes can be omitted.

[0029] 3-1. Step S11: Example of the drawing area display step In this step, the drawing area display unit 320 displays the first drawing area Rg1 shown in Fig. 4A. The first drawing area Rg1 can be drawn by receiving input from the input unit (an electronic pen in this embodiment). The first drawing area Rg1 has six-sided areas (areas Rg11 to Rg16) corresponding to the six-sided drawing in cube map projection.

[0030] In FIG. 4A, an area Rg11 corresponds to the upper surface area that can be seen when looking upward from an arbitrary viewpoint (an arbitrary fixed point) in the VR space. The region Rg12 corresponds to the lower surface region that can be seen when looking downward from an arbitrary viewpoint (an arbitrary fixed point) in the VR space. The region Rg13 corresponds to the front region that can be seen when looking forward from an arbitrary viewpoint (an arbitrary fixed point) in the VR space. The region Rg14 corresponds to the rear region that can be seen when looking backward from an arbitrary viewpoint (an arbitrary fixed point) in the VR space. Area Rg15 corresponds to the left surface area that can be seen when looking left from any viewpoint (any fixed point) in the VR space. Area Rg16 corresponds to the right surface area that can be seen when looking to the right from any viewpoint (any fixed point) in the VR space. Each of the regions (regions Rg11 to Rg16) is rectangular, and in this embodiment is square, and each of the regions (regions Rg11 to Rg16) has the same size. Each region (region Rg11 to region Rg16) has four edges (edge ​​fu, edge fb, edge fr, and edge fl), and these edges have a connection relationship with adjacent regions. For example, edge fu of region Rg13 has a connection relationship with edge fb of region Rg11, so that the depicted pictures are continuous. Furthermore, edge fu of region Rg16 has a connection relationship with edge fr of region Rg11, and this connection relationship is schematically indicated by arrow Ar in FIG. 4A. The remaining five arrows shown in FIG. 4A also indicate the connection relationship between the edges of both regions. For the positional relationship of the images depicted in each region, please also refer to FIG. 11B.

[0031] In the six-sided region of the first rendering region Rg1, first auxiliary lines ad1 for rendering using cube map projection are displayed, as shown by the solid lines in Fig. 4A. The first auxiliary lines ad1 have auxiliary lines displayed to separate each region (regions Rg11 to Rg16). Although not shown in FIG. 4A, the first auxiliary line ad1 may have grid lines that are displayed in each region (regions Rg11 to Rg16) and that make it easier for the user to draw pictures in each region in accordance with cube map projection. The grid lines may be displayed transparently above the drawn picture so that the user can easily see the drawn picture. This allows both the grid lines and the drawn picture to be visually recognized at the same time. The grid lines may be lines that correspond to the unique longitude and latitude lines in cube map projection, and the shape of the lines is not particularly limited.

[0032] In this step, the rendering area display unit 320 also displays the rectangular second rendering area Rg2 shown in FIG. 4B. It is preferable that the first drawing area Rg1 and the second drawing area Rg2 are displayed by tapping (selecting) a specific object on the display unit of the user terminal 3. For example, when the user taps an object (icon) corresponding to the first drawing area Rg1, the first drawing area Rg1 is displayed. Also, when the user wishes to display the second drawing area Rg2, the user taps an object (icon) corresponding to the second drawing area Rg2, and the first drawing area Rg1 is closed and the second drawing area Rg2 is displayed.

[0033] The second rendering area Rg2 can be rendered by receiving input from an input unit (an electronic pen in this embodiment). The second rendering area Rg2 has each area (area Rg21 to area Rg26) in equirectangular projection.

[0034] In Figure 4B, region Rg21 corresponds to the upper surface region that can be seen when looking upward from any viewpoint (any fixed point) in the VR space. Region Rg21 also corresponds to region Rg11 of the first rendering region Rg1 described above. Region Rg22 corresponds to the bottom surface region that can be seen when looking downward from any viewpoint (any fixed point) in VR space. Region Rg22 also corresponds to region Rg12 of the first rendering region Rg1 described above. Region Rg23 corresponds to the front region that can be seen when looking forward from any viewpoint (any fixed point) in the VR space. Region Rg23 also corresponds to region Rg13 of the first rendering region Rg1 described above. Region Rg24 corresponds to the rear region that can be seen when looking backward from any viewpoint (any fixed point) in the VR space. Region Rg24 also corresponds to region Rg14 of the first rendering region Rg1 described above. Region Rg25 corresponds to the left surface region that can be seen when looking left from any viewpoint (any fixed point) in the VR space. Region Rg25 also corresponds to region Rg15 of the first rendering region Rg1 described above. Region Rg26 corresponds to the right surface region that can be seen when looking to the right from any viewpoint (any fixed point) in the VR space. Region Rg26 also corresponds to region Rg16 of the first rendering region Rg1 described above. Each region (region Rg21 to region Rg26) has a shape that corresponds to equirectangular projection. Region Rg21 and region Rg22 have the same shape but are symmetrical from top to bottom, and also have the same size. Regions Rg23 to Rg26 have the same shape and are also the same size.

[0035] In the second rendering area Rg2, second auxiliary lines ad2 for rendering using equirectangular projection are displayed, as shown in Fig. 4B. The second auxiliary lines ad2 include auxiliary lines (see dashed lines) displayed to separate each area (areas Rg11 to Rg16) and rectangular auxiliary lines (solid lines) that define the outer edge of the second rendering area Rg2. Although not shown in FIG. 4B, the second auxiliary lines ad2 may have grid lines that are displayed in each region (regions Rg21 to Rg26) and that make it easier for the user to draw pictures in accordance with equirectangular projection in each region. The grid lines may be displayed transparently above the drawn picture so that the user can easily see the drawn picture. This allows both the grid lines and the drawn picture to be viewed simultaneously. The grid lines may be lines that correspond to the unique longitude and latitude lines in equirectangular projection, and the shape of the lines is not particularly limited.

[0036] Although not shown, in the embodiment, a third rendering area for rendering a third image D3 (FIGS. 9A to 9C) can also be displayed. In one example of the embodiment, the third image D3 is an animated image using planar projection. The third image D3 is superimposed on a composite image (the converted first image D11 and the second image D2) shown using equirectangular projection, which will be described later.

[0037] 3-2. Step S12 In this step, the control unit 32 makes the displayed drawing areas (first drawing area Rg1, second drawing area Rg2, third drawing area) ready to accept input from an input unit such as an electronic pen. At this time, the user can freely draw pictures in the drawing areas using the electronic pen, and the user terminal 3 (tablet terminal) instantly records the input and reflects it on the screen in real time. In addition, the user terminal 3 may display, for example, a color palette and toolbar, allowing the user to change the color and thickness of the pen while drawing. There may also be an eraser tool or the like to correct incorrect drawings. Note that the drawing interface is not limited to this, and various known methods may be employed.

[0038] The image depicted in the first depiction area Rg1 is the first image D1, the image depicted in the second depiction area Rg2 is the second image D2, and the image depicted in the third depiction area is the third image D3.

[0039] 3-2-1. About the first image The first image D1 is an image represented using cube map projection. In one embodiment, the first image D1 is an image of a tree or other complex shape created using cube map projection. As shown in FIG. 4A, cube map projection allows for a three-dimensional representation to be realized on a plane by rendering an image in an area divided into six faces of a cube. As shown in FIGS. 5 to 6F, cube map projection is suitable for rendering objects with complex shapes, such as tree branches and the details of buildings.

[0040] 3-2-2. About the second image The second image D2 is an image represented using equirectangular projection. In an example embodiment, the second image D2 is an image of the sky, clouds, or the like created using equirectangular projection. Equirectangular projection is particularly suitable for depicting elements that vary gradually or without change over a wide area, such as a simple, nearly solid image like the sky, or a monotonously colored image like clouds. This is because equirectangular projection has significant distortion of latitude and longitude lines, making it difficult to accurately depict complex shapes by hand. If a picture with a complex shape is depicted in the second depiction area Rg2 corresponding to equirectangular projection, and the equirectangular projection image (the image depicted in the second depiction area Rg2) is converted into a spherical image, processing is performed to connect the boundary portions of the image (the lower and upper portions of FIG. 11A ), but the connection of the boundary portions will be distorted.

[0041] In contrast, the cube map projection described above is less affected by such distortion than the equirectangular projection, and therefore can depict an image in a state similar to that of the planar projection. For example, if the upper part is depicted in region Rg11 (see FIG. 4A) and the edges of the regions connected to region Rg11 (regions Rg13 to Rg16) are properly drawn, it is possible to avoid visually perceiving distortion. As shown in FIG. 12A (an image looking up at a spherical image), the inventors have confirmed that distortion can be avoided. Similarly, if the lower part is depicted in region Rg12 (see FIG. 4A) and the edges of regions connected to region Rg12 (regions Rg13 to Rg16) are properly drawn, it is possible to prevent distortion from being visible. As shown in FIG. 12B (an image looking upward in a spherical image), the inventors have confirmed that distortion can be prevented. However, as shown in FIG. 4A, since the regions are connected at their edges, it is important to be aware of the edges where the regions are connected when depicting them.

[0042] In step S12, restrictions may be placed on the pixel values ​​of the picture that can be rendered as the second image D2. For example, in the horizontal direction of the second rendering region Rg2, when each of the RGB colors is expressed in 256 gradations (0 to 255), the second rendering region Rg2 is preferably restricted so that the R value (red) is greater than or equal to a lower limit r1 and less than or equal to an upper limit r2, the G value is greater than or equal to a lower limit g1 and less than or equal to an upper limit g2, and the B value is greater than or equal to a lower limit b1 and less than or equal to an upper limit b2.

[0043] The lower limit value r1 is specifically, for example, 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, or 120, and may be within a range between any two of the numerical values ​​exemplified here. The lower limit value g1 is specifically, for example, 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, or 120, and may be within a range between any two of the numerical values ​​exemplified here. The lower limit b1 is specifically, for example, 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, or 120, and may be within a range between any two of the numerical values ​​exemplified here.

[0044] The upper limit value r2 is, for example, 130, 140, 150, 160, 170, 180, 190, 200, 210, 220, 230, 240, 250, or 255, and may be within a range between any two of the values ​​exemplified here. Note that the upper limit value r2 is greater than the lower limit value r1. The upper limit value g2 is, for example, 130, 140, 150, 160, 170, 180, 190, 200, 210, 220, 230, 240, 250, or 255, and may be within a range between any two of the values ​​exemplified here. Note that the upper limit value g2 is greater than the lower limit value g1. Specifically, the upper limit value b2 is, for example, 130, 140, 150, 160, 170, 180, 190, 200, 210, 220, 230, 240, 250, or 255, and may be within a range between any two of the numerical values ​​exemplified here. Note that the upper limit value b2 is greater than the lower limit value b1.

[0045] In addition, in order to maintain a monotonous color change, the range of variation in RGB gradation between adjacent pixels may be limited to a certain value or less. It is possible to provide a function to notify a warning when the pixel values ​​of the picture that can be depicted as the second image D2 deviate from the limits. In this way, consistency and quality of the depicted images can be ensured.

[0046] 3-2-3. About the third image The third image D3 is a picture image rendered using a planar projection. In other words, the third image D3 is an image created using conventional animation techniques and is a picture image rendered in a planar orthogonal coordinate system. In other words, the term "planar projection" here refers to an image rendered in a format without depth or distortion based on a general two-dimensional coordinate system. Unlike cube map projection and equirectangular projection, this format does not involve deformation based on a specific projection. While cube map projection and equirectangular projection are also rendered on a two-dimensional plane as shown in FIGS. 4A and 4B, they involve special coordinate transformations that project three-dimensional space onto a two-dimensional plane. In contrast, planar projection is distinguished as a format that uses a simple planar orthogonal coordinate system and does not introduce distortion due to projection. Thus, in the embodiments, the term "planar projection" refers to an image rendered in a simple planar orthogonal coordinate system and is clearly distinguished from special projection methods such as cube map projection and equirectangular projection. 9A to 9C, the third image D3 is configured with animation images D31 to D33. By using the third image D3, the information processing system 100 can incorporate dynamic elements into the spherical image, thereby increasing the visual impact and further enhancing the appeal of the spherical image.

[0047] 3-3. Step S13: Example of Acquisition Step In this step, the control unit 32 determines whether rendering in the first rendering area Rg1, the second rendering area Rg2, and the third rendering area (not shown) is complete. For example, the user can indicate to the control unit 32 that rendering is complete by selecting an object (icon) (not shown) indicating that rendering is complete. When it is determined that rendering is complete, the image acquisition unit 321 acquires (reads from the storage unit 31) the first image D1 rendered in the first rendering area Rg1 and the second image D2 rendered in the second rendering area Rg2. That is, in this step, the image rendered in the first rendering area Rg1 is acquired (read) as the first image D1, the image rendered in the second rendering area Rg2 is acquired (read) as the second image D2, and the image rendered in the third rendering area is acquired (read) as the third image D3.

[0048] 3-4. Step S14: Example of Image Conversion Step In this step, the image conversion unit 322 converts the first image D1 acquired in step S13 from the arrangement of the cube map projection to the arrangement of the equirectangular projection to generate a converted first image D11 (see FIG. 7). The method of converting an image from the cube map projection to the equirectangular projection is not particularly limited, and may be realized, for example, by executing image processing as described below.

[0049] When converting the first image D1 from a cube map projection layout to an equirectangular projection layout, the image conversion unit 322 first converts the six-plane image data represented in the cube map projection into spherical coordinates. Specifically, the image conversion unit 322 acquires position information for each pixel in each region and maps it from cubic coordinates to a spherical coordinate system. Next, the image conversion unit 322 rearranges the acquired spherical coordinate data based on the equirectangular projection and projects it onto a two-dimensional plane based on the corresponding longitude and latitude information. During this process, the image conversion unit 322 appropriately corrects each pixel value through interpolation processing to maintain boundary continuity. Furthermore, the image conversion unit 322 may apply a correction algorithm to areas where distortion occurs, for example, to ensure smooth and accurate reproduction of the converted image. This prevents distortion of depiction elements containing fine details, such as tree branches and leaves, when converting from cube map projection to equirectangular projection, thereby improving the fidelity of the rendered content. After these processes, the image conversion unit 322 outputs (generates) a converted first image D11 according to equirectangular projection.

[0050] Furthermore, in the embodiment, since not only the first image D1 and the second image D2 but also the third image D3 (FIGS. 9A to 9C) is used, it is preferable to perform image processing on the third image D3 as well, converting the layout from planar projection to equirectangular projection. This image processing is not particularly limited, and can be realized, for example, as described below. That is, the image conversion unit 322 generates a converted third image (not shown) from the third image D3.

[0051] To convert an image represented in a planar projection to an equirectangular projection, first map the planar projection image data to spherical coordinates. Specifically, each pixel position in the planar projection image is projected to the corresponding spherical coordinates. This projection involves recalculating the pixel positional relationships into spherical coordinates based on the center point of the image. The resulting spherical coordinate data is then relocated based on the equirectangular projection and projected onto a two-dimensional plane according to the corresponding longitude and latitude information. Note that, as with the conversion from a cubemap projection layout to an equirectangular projection layout, the aforementioned interpolation processes and correction algorithms may be implemented as appropriate.

[0052] 3-5. Step S15: Example of Image Synthesis Step In this step, the image composition unit 323 generates a composite image by combining at least the converted first image D11 and the second image D2. In the embodiment, the image composition unit 323 generates a composite image by combining the converted first image D11, the second image D2, and the converted third image. At this time, the converted first image D11 is superimposed on the second image D2 to generate the composite image. In the embodiment, the converted first image D11 is a hand-drawn picture of a complex shape such as a tree, and this is placed on the second image D2, which is a solid or nearly solid picture of the sky. Therefore, pixel areas of the second image D2 where the converted first image D11 is drawn are hidden by the pixels of the converted first image D11, and pixel areas of the second image D2 where the converted first image D11 is not drawn are not hidden by the converted first image D11 and are perceived as sky. More specifically, in one embodiment, the second image D2 is placed in the bottom layer, the converted first image D11 is placed in the middle layer, and the converted third image is placed in the top layer, and these are overlaid to generate a composite image (not shown). Here, the superimposed image (the image below) is described as being hidden by the image it overlays (the image above), but this is not necessarily limited to this, and the superimposed image may be transparent due to image transparency processing.

[0053] In the embodiment, the third image D3 is composed of images corresponding to a plurality of frames, namely animation images D31 to D33. Therefore, it is preferable that composite images are also generated individually, such as a composite image having animation image D31, a composite image having animation image D32, and a composite image having animation image D33.

[0054] Furthermore, when both the upper and lower images are depicted at a pixel at any coordinate in the composite image, the pixel in the upper layer is the pixel that is visible, unless the pixel in the upper layer has information such as a pixel with transparency. Furthermore, if pixels of the lower image are present in an area that is not depicted in the upper image, the pixels of the lower image will be visually recognized as a composite image.

[0055] Furthermore, in generating the composite image in this step, a process of adjusting the transparency and color tone of each image may be performed. For example, the transparency of the second image D2 (sky part) may be set high so that the first image D11 and the third image D3 after conversion are more easily visible. Furthermore, since the third image D3 is a dynamically changing animation image, it is preferable that the appropriate position and size are calculated for each frame before being composited. It is preferable that there is an information processing step in which the position and size are designated by the user (for example, an instruction to that effect is received from the user immediately before the images are composited in this step). This step may also include color space conversion to maintain color harmony and blending to smooth the boundaries between the images to be superimposed, resulting in a more unified composite image.

[0056] The image composition unit 323 performs information processing on the composite image generated by combining the converted first image D11, the second image D2, and the converted third image, and finally generates a celestial sphere image (VR image). That is, the image composition unit 323 performs information processing to convert the superimposed composite image into spherical coordinates and output it in a format suitable for celestial sphere display. Note that the control unit 32 may display the composite image in equirectangular projection format generated by combining the converted first image D11, the second image D2, and the converted third image as a preview image before creating the celestial sphere image. This allows the user to grasp the outline of the composite image in advance.

[0057] 3-6. Step S16 The control unit 32 displays the composite image as a celestial sphere image on a desired display unit or device. The desired display unit is, for example, the display unit of the user terminal 3. In this case, the composite image can be viewed through a viewer that is an application for viewing celestial sphere images installed on the user terminal 3. With such a viewer, the viewing direction from a fixed point can be adjusted. Alternatively, the composite image can be displayed on a device such as VR goggles that can communicate with the user terminal 3 so that the user can view the image. In this way, in the embodiment, it is possible to view an immersive celestial sphere image based on a hand-drawn picture.

[0058] 4. Description of the Functions and Effects of the Embodiments In the information processing method according to the embodiment, an image suitable for representation using cube map projection can be used for the first image D1, and an image suitable for representation using equirectangular projection can be used for the second image D2, and it is possible to select the projection according to the content of each image, thereby improving the quality of the composite image. That is, cube map projection is used for the first image D1, and with this projection, distortion in each region (regions Rg11 to Rg16) is small, making it easy to draw by hand images that require fine details such as trees. In addition, the second image D2 uses equirectangular projection, which has the advantage that it can continuously represent a 360-degree field of view in the horizontal direction as a single rectangular image, making it suitable for seamlessly representing monotonous textures such as the sky (uniform gradations or simple solid colors). In particular, monotonous patterns such as the sky are easy to depict because the whole image flows naturally.

[0059] 5. Variations For example, by shifting the horizontal positions of the right-eye image and the left-eye image by a few pixels, the sense of depth in human vision is reproduced. This technique is particularly effective for improving the three-dimensional effect of images in VR space. This modification improves on this point. In other words, this modification shifts the positions of the images seen by the left and right eyes to make the images appear more three-dimensional. This modification is suitable for displaying a composite image on VR goggles that display independent images, as different images are seen by the left and right eyes. In this modification, as shown in FIG. 13 , the control unit 32 further includes a three-dimensional image generation unit 324. The stereoscopic image generator 324 has a function of generating a composite image for the right eye and a composite image for the left eye, relatively changing the positional relationship for displaying these composite images, and generating binocular parallax, so that the composite images appear stereoscopic. In the modified example, steps S16 and S17 are newly added, as shown in FIG.

[0060] Step S16 is an example of a copy generating step in the stereoscopic image generating step, and step S17 is an example of an image parallax forming step in the stereoscopic image generating step.

[0061] In step S16, the stereoscopic image generator 324 generates two identical composite images. That is, it generates copies of the composite image generated in step S15 described in the embodiment. Here, of the two identical composite images, one composite image is an image for right-eye display, and the other composite image is an image for left-eye display.

[0062] In step S17, the stereoscopic image generator 324 executes a process of relatively differentiating the display position of one composite image (for right eye display) from the display position of the other composite image (for left eye display). Specifically, for example, the process of differentiating the display positions of both composite images is executed. Then, in step S18, the controller 32 displays both composite images as a celestial sphere image on a desired display unit or device.

[0063] Various embodiments are exemplified below, and the embodiments shown below can be combined with each other. [Appendix 1] An information processing method for generating a composite image by synthesizing handwritten images, comprising: The method includes an image acquisition step, an image conversion step, and an image synthesis step, In the image acquisition step, a first image and a second image are acquired, the first image is an image represented in cube map projection; the second image is an image represented in equirectangular projection; In the image conversion step, the first image is converted from the arrangement of the cube map projection to the arrangement of the equirectangular projection to generate a converted first image; In the image synthesis step, the synthesized image is generated by synthesizing at least the transformed first image and the second image. [Appendix 2] 10. The information processing method according to claim 1, In the image synthesis step, the transformed first image is superimposed on the second image to generate the synthesized image. [Appendix 3] 10. The information processing method according to claim 1 or 2, The image acquisition step further acquires a third image, the third image is a planar projection image; In the image conversion step, the third image is converted from the layout of the planar projection to the layout of the equirectangular projection to generate a converted third image; In the image synthesis step, the transformed first image, the second image, and the transformed third image are synthesized to generate the synthesized image. [Appendix 4] 4. The information processing method according to claim 3, In the image synthesis step, the synthesized image is generated by superimposing the transformed third image on the transformed first image and the transformed second image. [Appendix 5] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 4, A depiction area display step is further provided, In the drawing area display step, a first drawing area is displayed, the first rendering area can be rendered by receiving input from an input unit, and a six-sided area corresponding to a six-sided drawing in the cube map projection is displayed in the first rendering area; a first auxiliary line for depicting the six-sided area using the cube map projection is displayed; An information processing method, wherein in the image acquisition step, an image depicted in the first depiction area is acquired as the first image. [Appendix 6] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 5, A depiction area display step is further provided, In the drawing area display step, a rectangular second drawing area is displayed, the second rendering area can be rendered by receiving input from an input unit, and second auxiliary lines for rendering in the equirectangular projection are displayed in the second rendering area; An information processing method, wherein in the image acquisition step, an image depicted in the second depiction area is acquired as the second image. [Appendix 7] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 6, Further comprising a stereoscopic image generating step, the stereoscopic image generating step includes a copy generating step and an image parallax forming step, The copy generating step generates two identical composite images; one of the two identical composite images is an image for right-eye display and the other composite image is an image for left-eye display; In the image parallax forming step, a process is executed to relatively change a display position of one of the composite images and a display position of the other of the composite images. [Appendix 8] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 7, The information processing method, wherein the composite image is a spherical image. [Appendix 9] A program that causes a computer to execute the information processing method according to any one of Supplementary Note 1 to Supplementary Note 8. [Appendix 10] An information processing device that executes each step of the information processing method according to any one of Supplementary notes 1 to 8. [Explanation of symbols]

[0064] 1: Information processing server 2: Communication network 3: User terminal 10: Communications Department 11: Storage section 12: Control section 13: Output section 14: Input section 15: Communication bus 30: Communications Department 31: Storage section 32: Control section 33: Output section 34: Input section 35: Communication bus 100: Information Processing Systems 320:Description area display area 321: Image acquisition unit 322: Image conversion unit 323: Image synthesis unit 324: 3D image generation unit

Claims

1. An information processing method for generating a composite image by synthesizing handwritten images, comprising: The method includes an image acquisition step, an image conversion step, and an image synthesis step, In the image acquisition step, a first image and a second image are acquired; the first image is an image represented in cube map projection; the second image is an image represented in equirectangular projection; In the image conversion step, the first image is converted from the arrangement of the cube map projection to the arrangement of the equirectangular projection to generate a converted first image; In the image synthesis step, at least the transformed first image and the second image are synthesized to generate the synthesized image.

2. 2. The information processing method according to claim 1, In the image synthesis step, the transformed first image is superimposed on the second image to generate the synthesized image.

3. 3. The information processing method according to claim 1 or 2, The image acquisition step further acquires a third image; the third image is a planar projection image; In the image conversion step, the third image is converted from the layout of the planar projection to the layout of the equirectangular projection to generate a converted third image; In the image synthesis step, the transformed first image, the second image, and the transformed third image are synthesized to generate the synthesized image.

4. 4. The information processing method according to claim 3, In the image synthesis step, the synthesized image is generated by superimposing the transformed third image on the transformed first image and the transformed second image.

5. 3. The information processing method according to claim 1 or 2, A depiction area display step is further provided, In the depiction area display step, a first depiction area is displayed, the first rendering area can be rendered by receiving an input from an input unit, and a six-sided area corresponding to a six-sided view in the cube map projection is displayed in the first rendering area; a first auxiliary line for depicting the six-sided area using the cube map projection method is displayed in the six-sided area; An information processing method, wherein in the image acquisition step, an image depicted in the first depiction area is acquired as the first image.

6. 3. The information processing method according to claim 1 or 2, A depiction area display step is further provided, In the drawing area display step, a rectangular second drawing area is displayed, the second rendering area can be rendered by receiving input from an input unit, and second auxiliary lines for rendering in the equirectangular projection are displayed in the second rendering area; An information processing method, wherein in the image acquisition step, an image depicted in the second depiction area is acquired as the second image.

7. 3. The information processing method according to claim 1 or 2, Further comprising a stereoscopic image generating step, the stereoscopic image generating step includes a copy generating step and an image parallax forming step, The copy generating step generates two identical composite images; one of the two identical composite images is an image for right-eye display and the other composite image is an image for left-eye display; In the image parallax forming step, a process is executed to relatively change a display position of one of the composite images and a display position of the other of the composite images.

8. 3. The information processing method according to claim 1 or 2, The information processing method, wherein the composite image is a spherical image.

9. A program that causes a computer to execute the information processing method according to claim 1 or 2.

10. An information processing device that executes each step of the information processing method according to claim 1 or 2.

Images