Image generation method, image generation system, and program

Abstract

To enable a user not having expertise to easily perform projection mapping.SOLUTION: An information processing apparatus 20A causes a display device 420 to display a superimposed image obtained by superimposing a material image applied with transmission processing on a captured image obtained by capturing a projection target object SC, which is a projection target of an image from a projector 10, with a camera 30, and a user interface image for receiving an input for determining a position of the camera 30. The information processing apparatus 20A generates projection image data representing a projection image by correcting the material image according to a three-dimensional shape measured on the basis of the captured image obtained by imaging, from the determined position, with the camera 30, the projection target object SC onto which a pattern image for three-dimensional measurement is projected from the projector 10, and outputs the projection image data to the projector 10.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image generation method, an image generation system, and a program. [Background technology]

[0002] In recent years, projection mapping, which creates various effects by projecting various images from a projector onto a three-dimensional object, has become increasingly popular. Hereinafter, the object onto which the image is projected by the projector is referred to as the projection target. Projection mapping requires advance preparation, such as distorting the image projected by the projector to match the three-dimensional shape of the projection target. This is because the image reflected on the surface of the projection target will be distorted according to the three-dimensional shape of the projection target. Various technologies have been proposed to assist advance preparation for projection mapping, one example of which is the technology disclosed in Patent Document 1. In the technology disclosed in Patent Document 1, a correction image including markers for position detection is projected onto the projection target by a projector. The correction image reflected on the surface of the projection target is captured by a detection device, and the markers are detected from the captured correction image. By correcting the original image data based on the marker detection results, the image projected by the projector is corrected according to the shape of the projection target. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-168640 Summary of the Invention [Problem to be solved by the invention]

[0004] To properly apply distortion to an image projected from a projector according to the three-dimensional shape of the projection target, it is necessary for the markers to be accurately depicted in the correction image, without excess or deficiency. If the markers are not accurately depicted in the correction image, marker detection will be hindered. In other words, the technology disclosed in Patent Document 1 presupposes that the user has a certain level of specialized knowledge about projection mapping, such as specialized knowledge about imaging conditions such as the position and orientation of the detection device that captures the correction image. However, users who wish to perform projection mapping do not always have specialized knowledge about projection mapping. The technology disclosed in Patent Document 1 has the problem that users without specialized knowledge about projection mapping cannot easily perform projection mapping. [Means for solving the problem]

[0005] One aspect of the image generating method of the present disclosure includes displaying a superimposed image in which a first captured image obtained by capturing an image of a projection object using a camera in a real space in which a projector and the projection object onto which an image from the projector is projected are superimposed with a first image that has undergone transparency processing, and a user interface image that accepts input to determine the position of the camera in the real space; generating a second image by correcting the first image in accordance with the shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector using the camera from the position; and outputting image data representing the second image to the projector.

[0006] Furthermore, one aspect of the image generation system of the present disclosure includes a display device and a processing device that controls the display device, and the processing device performs the following operations: displaying on the display device a superimposed image in which a first image that has been subjected to transparency processing is superimposed on a first captured image obtained by capturing an image of a projection object using a camera in a real space in which a projector and a projection object onto which an image from the projector is projected are located; and displaying a user interface image that accepts input for determining the position of the camera in the real space; generating a second image by correcting the first image in accordance with the shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector from the position; and outputting image data representing the second image to the projector.

[0007] The program disclosed herein also causes a computer to perform the following operations: display on a display device a superimposed image in which a first captured image obtained by capturing an image of a projection object using a camera in a real space in which a projector and a projection object onto which an image from the projector is projected are superimposed with a first image that has undergone transparency processing; and a user interface image that accepts input to determine the position of the camera in the real space; generate a second image by correcting the first image in accordance with the shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector from the position using the camera; and output image data representing the second image to the projector. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an image generation system 1A according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram showing an example of an image represented by material data D1 in the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of an information processing device 20A. [Figure 4] FIG. 10 is a diagram illustrating an example of a superimposed image. [Figure 5] FIG. 10 is a diagram for explaining a superimposed image. [Figure 6] 5A to 5C are diagrams illustrating display examples of a superimposed image and a user interface image in the first embodiment. [Figure 7] 10A and 10B are diagrams for explaining the angle of view of a material image and the angle of view of a captured image. [Figure 8] 10A and 10B are diagrams illustrating an example of a superimposed image when the angle of view of a material image and the angle of view of a captured image match. [Figure 9] FIG. 2 is a diagram illustrating an example of projection mapping realized by the first embodiment. [Figure 10] 10 is a flowchart showing the flow of an image generating method executed by a processing device 210 of an information processing device 20A in accordance with a program PA. [Figure 11] FIG. 10 is a diagram illustrating an example of the configuration of an image generation system 1B according to a second embodiment of the present disclosure. [Figure 12] FIG. 10 is a diagram showing an example of an image represented by material data D1 in the second embodiment. [Figure 13] FIG. 2 is a diagram illustrating an example of the configuration of an information processing device 20B. [Figure 14] 10A and 10B are diagrams illustrating display examples of a superimposed image and a user interface image in the second embodiment. [Figure 15] FIG. 10 is a diagram illustrating an example of a mask image. [Figure 16] FIG. 10 is a diagram illustrating an example of a projection image generated using a mask image. [Figure 17] FIG. 10 is a diagram illustrating an example of projection mapping realized in a second embodiment. [Figure 18] 10 is a flowchart showing the flow of an image generating method executed by the processing device 210 of the information processing device 20B in accordance with the program PB. DETAILED DESCRIPTION OF THE INVENTION

[0009] The embodiments described below are subject to various technically preferable limitations, but the embodiments of the present disclosure are not limited to the following embodiments. 1. First embodiment FIG. 1 is a diagram showing an example configuration of an image generation system 1A according to a first embodiment of the present disclosure. The image generation system 1A is an information processing system that generates image data representing a projection image to be projected from a projector 10 onto a projection target SC in projection mapping. Hereinafter, the image data representing the projection image will be referred to as projection image data. As shown in FIG. 1, the image generation system 1A includes an information processing device 20A and a terminal device 40. In addition to the image generation system 1A, FIG. 1 also shows the projector 10, a camera 30, a projection target SC, a communication network 50, and a material management device 60.

[0010] The projection target SC in this embodiment is a mannequin that resembles the upper half of a human body and is wearing a white, patternless T-shirt. The projection target SC and the projector 10 are installed, for example, in the sales floor of a retail store that sells clothing. In this embodiment, a simulated product display is realized by projection mapping by projecting a projection image corresponding to the color and pattern of the T-shirt from the projector 10 onto the projection target SC. Hereinafter, the store where the projection target SC and the projector 10 are installed will be referred to as the implementation store.

[0011] The information processing device 20A is, for example, a stick-type personal computer. The information processing device 20A has a male connector that complies with a predetermined standard such as USB (Universal Serial Bus). The projector 10 has a female connector that corresponds to the male connector. By inserting the male connector of the information processing device 20A into the female connector of the projector 10, the information processing device 20A and the projector 10 are electrically connected. In addition, the information processing device 20A communicates with the camera 30 and the terminal device 40 wirelessly or via a wired connection.

[0012] The camera 30 is a device for capturing an image of the projection target SC. The camera 30 is installed in the implementation store using, for example, a tripod or the like with its optical axis facing the projection target SC. The camera 30 captures an image under the control of the information processing device 20A, and outputs image data representing the captured image to the information processing device 20A. Hereinafter, the image data representing the captured image will be referred to as captured image data.

[0013] The communication network 50 is, for example, a telecommunications line such as the Internet. A material management device 60 is connected to the communication network 50. The material management device 60 is, for example, a data server. One or more pieces of material data D1 are stored in advance in the material management device 60. The material data D1 is image data representing an image that serves as the basis for a projection image projected from the projector 10 onto the projection target SC. Hereinafter, an image represented by the material data will be referred to as a material image. A material image is an example of a first image in the present disclosure. FIG. 2 is a diagram showing an example of a material image GA1 in this embodiment. As shown in FIG. 2, the material image GA1 in this embodiment is an image of a T-shirt having a pattern. The material data D1 is created by a designer or the like in charge of designing the T-shirt and uploaded to the material management device 60. The material data D1 uploaded to the material management device 60 can be downloaded to the terminal device 40 by communication via the communication network 50.

[0014] The terminal device 40 is a smartphone used by a user of the projector 10. In this embodiment, the user of the projector 10 is a store clerk working at the store where the projector 10 is used. As shown in FIG. 1 , the terminal device 40 includes an external IF device 410, a display device 420, and an input device 430.

[0015] The external IF device 410 communicates with the material management device 60 via the communication network 50 and includes a communication circuit for communicating with the information processing device 20A. IF stands for Interface. The display device 420 includes a liquid crystal display and a drive circuit for the liquid crystal display. The terminal device 40 displays various images on the display device 420 under the control of the information processing device 20A. The input device 430 is a transparent sheet-like pressure sensor provided to cover the display area of ​​the display device 420 and accepts input operations from the user. The terminal device 40 transmits input operation data indicating the user's input operation on the input device 430 to the information processing device 20A via the external IF device 410. As a result, the user's input operation is transmitted to the information processing device 20A.

[0016] Although details will be described later, in response to an input operation made to the terminal device 40, the information processing device 20A captures an image using the camera 30, generates projection image data based on the captured image data acquired from the camera 30 and material data D1 downloaded to the terminal device 40, and outputs the generated projection image data to the projector 10. A projection image represented by the projection image data generated by the information processing device 20A is projected from the projector 10 onto the projection target SC, thereby realizing a product display using projection mapping.

[0017] FIG. 3 is a diagram showing an example of the configuration of the information processing device 20A. As shown in FIG. 3, the information processing device 20A includes a processing device 210, an external IF device 220, and a storage device 230. The processing device 210 includes a processor such as a CPU (Central Processing Unit), i.e., a computer. The processing device 210 may be configured with a single processor or multiple processors. The processing device 210 functions as the control center of the information processing device 20A by operating in accordance with a program PA stored in the storage device 230.

[0018] The external IF device 220 includes the male connector described above. When this male connector is inserted into the female connector of the projector 10 and the information processing device 20A and the projector 10 are electrically connected, the external IF device 220 outputs data or signals provided from the processing device 210 to the projector 10. The external IF device 220 also includes a communication circuit for communicating with the camera 30 or the terminal device 40.

[0019] The storage device 230 is a recording medium readable by the processing device 210. The storage device 230 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM).

[0020] A program PA that causes the processing device 210 to execute the image generating method of the present disclosure is stored in advance in the nonvolatile memory of the storage device 230. Furthermore, identification information D2 that uniquely identifies the material management device 60 in the communication network 50 is stored in advance in the nonvolatile memory of the storage device 230. A specific example of the identification information D2 is a communication address assigned to the material management device 60. The volatile memory of the storage device 230 is used by the processing device 210 as a work area when executing the program PA.

[0021] When the processing device 210 detects a connection between the information processing device 20A and the projector 10, it reads the program PA from the non-volatile memory to the volatile memory and starts executing the read program PA. When the processing device 210, operating in accordance with the program PA, detects a connection of the terminal device 40 and the camera 30 to the external IF device 410, it transmits the identification information D2 to the terminal device 40. As a result, the terminal device 40 acquires the identification information D2.

[0022] When the identification information D2 is acquired by the terminal device 40, the user accesses the material management device 60 using the identification information D2 and downloads material data D1 representing a desired material image from the material management device 60 to the terminal device 40, thereby acquiring the material data D1. When the processing device 210, operating in accordance with the program PA, detects that the terminal device 40 has acquired the material data D1, it functions as a display control unit 210a, a first notification unit 210b, a measurement unit 210c, a first generation unit 210d, and an output unit 210e shown in FIG. 3. The display control unit 210a, the first notification unit 210b, the measurement unit 210c, the first generation unit 210d, and the output unit 210e shown in FIG. 3 are software modules realized by operating the processing device 210 in accordance with the program PA. The functions of the display control unit 210a, the first notification unit 210b, the measurement unit 210c, the first generation unit 210d, and the output unit 210e are as follows.

[0023] The display control unit 210a causes the camera 30 to capture an image at a predetermined cycle, such as every 1 millisecond. In this embodiment, the camera 30 is installed in the store with its optical axis facing the projection target SC, and therefore captures the projection target SC. The captured image of the projection target SC captured by the camera 30 under the control of the display control unit 210a is an example of a first captured image in the present disclosure. The display control unit 210a acquires captured image data from the camera 30 each time the display control unit 210a causes the camera 30 to capture an image. Each time the display control unit 210a acquires captured image data, the display control unit 210a generates image data representing a superimposed image GA5 based on the material data D1 downloaded to the terminal device 40 and the acquired captured image data.

[0024] FIG. 4 is a diagram illustrating an example of the superimposed image GA5. FIG. 5 is a diagram illustrating the superimposed image GA5. As shown in FIG. 5, the superimposed image GA5 is generated by superimposing an image GA3 obtained by performing a transparency process on a material image GA1 represented by material data D1 on a captured image GA4 represented by captured image data. The transparency process is a process of changing the transmittance of the material image GA1 to a value greater than 0% but less than 100%. The closer the transmittance is to 0%, the less light is transmitted, and the closer the transmittance is to 100%, the more light is transmitted. The material image GA1 that has undergone the transparency process becomes translucent. In the example shown in FIG. 4, the outline of the T-shirt in the material image GA1 and the outline of the pattern applied to the T-shirt are drawn with dotted lines, thereby expressing the translucency of the material image GA1. Because the material image GA1 is translucent in the superimposed image GA5, the user can view the captured image GA4 through the translucent material image GA1. The transmittance in the transparency processing may be any transmittance that allows the captured image GA4 to be seen through the material image GA1, and may be in the range of 10% to 90%, for example. The transmittance may be adjustable by the user using a user interface (not shown).

[0025] The display control unit 210a provides the generated image data to the terminal device 40, and causes the display device 420 to display the superimposed image GA5. The display control unit 210a also causes the display device 420 to display a user interface image GA6, which accepts an input operation to determine the position of the camera 30 in the store, i.e., the position of the camera 30 in real space, together with the superimposed image GA5. FIG. 6 is a diagram showing a display example of the superimposed image GA5 and the user interface image GA6. The user interface image GA6 in the example shown in FIG. 6 is an image of a virtual control that accepts a user's touch operation.

[0026] The first notification unit 210b outputs a notification urging the user to determine, as the position of the camera 30, a position where the angle of view of the material image GA1 in the superimposed image GA5 and the angle of view of the captured image GA4 in the superimposed image GA5 match. In this embodiment, the first notification unit 210b displays a message M1 on the display device 420, saying, "Move the camera to a position where the mannequin and the product overlap and press the 'OK' button," as shown in Fig. 6. In this embodiment, the notification is made by displaying the message M1, but the notification may also be made by outputting a sound representing the message M1.

[0027] FIG. 7 shows an example of the angle of view of material image GA1 in superimposed image GA5 and the angle of view of captured image GA4 in superimposed image GA5. The angle of view of material image GA1 refers to the field of view of the camera when capturing material image GA1. Note that material image GA1 may be an image of actual clothing, or may be a created image. If material image GA1 is not an image of actual clothing, the field of view of a virtual camera that represents how the clothing appears in material image GA1 corresponds to the angle of view of material image GA1. The angle of view of captured image GA4 refers to the field of view of camera 30 when capturing captured image GA4. The angle of view is determined according to the position of the camera and the direction of the optical axis. In FIG. 7, the angle of view of material image GA1 is depicted with a dotted line, and the angle of view of captured image GA4 is depicted with a dashed line. 7, the position of the camera 30 when capturing the captured image GA4 is position P1, and when the camera 30 moves to position P2, the angle of view of the material image GA1 and the angle of view of the captured image GA4 approximately match. That is, in the example shown in Fig. 7, position P2 is the position of the camera 30 where the angle of view of the material image GA1 in the superimposed image GA5 and the angle of view of the captured image GA4 in the superimposed image GA5 match.

[0028] The user who has viewed the message M1 can see the angle of view of the material image GA1 and the captured image through the superimposed image GA5 displayed on the display device 420 of the terminal device 40. GA4The user moves with the camera 30 while checking the angle of view of the material image GA1 and the captured image GA4. As the camera 30 moves, the angle of view of the captured image GA4 included in the superimposed image GA5 displayed on the display device 420 changes. For example, when the camera 30 moves to position P2, the superimposed image GA5 shown in FIG. 8 is displayed on the display device 420. When the angle of view of the material image GA1 and the angle of view of the captured image GA4 match, the user stops moving the camera 30 and performs a touch operation on the user interface image GA6.

[0029] When the measurement unit 210c receives a touch operation on the user interface image GA6, it performs 3D measurement to measure the shape of the projection target SC. More specifically, the measurement unit 210c outputs a signal to the projector 10 instructing it to project a series of pattern images for measuring the three-dimensional shape of the object. Specific examples of pattern images include a pattern image that encodes space, such as an image representing a Gray code pattern, or a pattern image representing a sine wave pattern. The measurement unit 210c also outputs a signal instructing the camera 30 to capture an image in synchronization with the output of a signal instructing the projection of each pattern image, i.e., with a predetermined delay from the output of the signal instructing the projection of the pattern image. As a result, the projection target SC, onto which the series of pattern images are projected from the projector 10, is captured for each pattern image by the camera 30 located at a position determined by the touch operation. In the example shown in FIG. 7, a touch operation is performed at position P2, and therefore, the camera 30 located at position P2 captures the image.

[0030] The measurement unit 210c generates conversion data for converting between the camera coordinate system and the world coordinate system based on a series of captured images obtained by capturing, for each pattern image, a projection target SC on which a series of pattern images are projected from the projector 10, using the camera 30 from a position determined by the touch operation. In other words, the measurement unit 210c measures the shape of the projection target SC based on the captured images obtained by capturing, for each pattern image, a projection target SC on which a series of pattern images are projected from the projector 10. Generating the conversion data corresponds to measuring the shape of the projection target SC. The series of captured images obtained by capturing, for each pattern image, a projection target SC on which a series of pattern images are projected from the projector 10, using the camera 30 from a position determined by the touch operation is an example of a second captured image in the present disclosure. The camera coordinate system is a two-dimensional coordinate system that defines a position within the captured image of the camera. The world coordinate system is a three-dimensional coordinate system that defines a position in real space. As for a specific algorithm for generating conversion data from a series of pattern images, an existing algorithm may be used appropriately depending on the type of pattern image.

[0031] The first generation unit 210d generates projection image data by performing coordinate transformation indicated by the transformation data on the material image GA1 represented by the material data D1. Performing coordinate transformation indicated by the transformation data on the material image GA1 corresponds to correcting the material image GA1 according to the shape of the projection target SC. Then, the output unit 210e outputs the projection image data to the projector 10. The projector 10 projects the projection image represented by the projection image data output from the information processing device 20A onto the projection target SC. The projection image is an example of a second image in the present disclosure.

[0032] In this embodiment, the camera 30 is positioned at a position where the angle of view of the captured image GA4 and the angle of view of the material image GA1 match. The fact that the angle of view of the captured image GA4 and the angle of view of the material image GA1 match means that the position of the camera relative to the subject when the material image GA1 was captured is approximately the same as the position of the camera 30 relative to the projection target SC, and that the direction of the optical axis of the camera when the material image GA1 was captured is approximately the same as the direction of the optical axis of the camera 30. Therefore, the fact that the angle of view of the captured image GA4 and the angle of view of the material image GA1 match means that the camera coordinate system for the camera 30 and the camera coordinate system of the camera that captured the material image are approximately the same. Since the camera coordinate system for the camera 30 is converted into a world coordinate system based on the conversion data, the camera coordinate system of the camera that captured the material image GA1 is also converted into a world coordinate system based on the conversion data. Since the converted material image GA1 is given a distortion according to the shape of the projection target SC, as shown in FIG. 9 , the material image GA1 is projected onto the surface of the projection target SC with almost no distortion.

[0033] Furthermore, the processing device 210 operating in accordance with the program PA executes the image generation method shown in Fig. 10. As shown in Fig. 10, the image generation method of this embodiment includes a display control process SA110, a first notification process SA120, a measurement process SA130, a first generation process SA140, and an output process SA150.

[0034] In the display control processing SA110, the processing device 210 functions as a display control unit 210a. In the display control processing SA110, the processing device 210 causes the display device 420 to display a superimposed image GA5 based on the material data D1 downloaded to the terminal device 40 and the captured image data acquired from the camera 30, and also causes the display device 420 to display a user interface image GA6.

[0035] In the first notification process SA120, the processing device 210 functions as the first notification unit 210b. In the first notification process SA120, the processing device 210 displays a message M1 on the display device 420, prompting the user to determine, as the position of the camera 30, a position where the angle of view of the material image GA1 and the angle of view of the captured image GA4 match.

[0036] In the measurement process SA130, the processing device 210 functions as a measurement unit 210c. In the measurement process SA130, the processing device 210 performs three-dimensional measurement to measure the shape of the projection target SC, and generates conversion data for converting between the camera coordinate system of the camera 30 and the world coordinate system.

[0037] In the first generation process SA140, the processing device 210 functions as a first generation unit 210d. In the first generation process SA140, the processing device 210 generates projection image data by performing coordinate transformation indicated by the transformation data generated in the measurement process SA130 on the material data D1 downloaded to the terminal device 40.

[0038] In the output process SA150, the processing device 210 functions as an output unit 210e. In the output process SA150, the processing device 210 outputs the projection image data generated in the first generation process SA140 to the projector 10. The projector 10 projects a projection image represented by the projection image data output from the information processing device 20A onto the projection target SC. By projecting the image represented by the projection image data from the projector 10 onto the projection target SC, as shown in FIG. 9, the material image GA1 is projected without distortion on the surface of the projection target SC.

[0039] It should be noted here that all that is required of the user of projector 10 is to download material data D1 and to determine the position of camera 30 in response to message M1 and perform a touch operation, and no specialized knowledge of projection mapping is required. As described above, according to this embodiment, even if a retail store clerk does not have specialized knowledge of projection mapping, they can easily perform product displays using projection mapping.

[0040] In addition, when product displays related to clothing are realized using projection mapping, there is no need to prepare product samples for each color and pattern of the product for product display. Conventionally, retail stores and the like change their product lineup according to the change of seasons, and after the product lineup is changed, the previous product samples are no longer needed. Unnecessary product samples are sometimes sold at low prices, but most of them are discarded. Disposal of such product samples has been a problem from the perspective of effective use of resources. According to this embodiment, waste of resources is reduced.

[0041] 2. Second embodiment FIG. 11 is a diagram illustrating an example configuration of an image generation system 1B according to a second embodiment of the present disclosure. In FIG. 11, the same components as those in FIG. 1 are denoted by the same reference numerals. Similar to FIG. 1, FIG. 11 also illustrates a projection target SC, a projector 10, a camera 30, a communication network 50, and a material management device 60 in addition to the image generation system 1B. Comparing FIG. 11 with FIG. 1, it is clear that the configuration of the image generation system 1B differs from that of the image generation system 1A in that it includes an information processing device 20B instead of the information processing device 20A. That is, the image generation system 1B includes an information processing device 20B and a terminal device 40. In this embodiment, the material data D1 stored in the material management device 60 is image data representing an image representing a pattern, such as a fabric or wallpaper, an image of an animal or person, or a landscape image. In this embodiment, the material image GB1 represented by the material data D1 is a landscape image including multiple mountains, clouds floating in the sky, and the sun, as shown in FIG. 12.

[0042] The information processing device 20B is a stick-type personal computer, similar to the information processing device 20A. FIG. 13 is a diagram illustrating an example of the configuration of the information processing device 20B. In FIG. 13, the same components as those in FIG. 3 are denoted by the same reference numerals. As is clear from a comparison of FIG. 13 with FIG. 3, the hardware configuration of the information processing device 20B is identical to that of the information processing device 20A. That is, the information processing device 20B includes a processing device 210, an external IF device 220, and a storage device 230. The configuration of the information processing device 20B differs from that of the information processing device 20A in that a program PB is stored in the storage device 230 instead of a program PA. The processing device 210 operating in accordance with the program PB functions as a display control unit 210a, a second notification unit 210f, a measurement unit 210c, a second generation unit 210g, and an output unit 210e, as shown in FIG. 13.

[0043] The information processing device 20B differs from the information processing device 20A in that a second notification unit 210f replaces the first notification unit 210b and a second generation unit 210g replaces the first generation unit 210d. However, in this embodiment, because the material image GB1 is a landscape image, the superimposed image GB2 that the display control unit 210a displays on the display device 420 is different from the superimposed image GA5 in the first embodiment. The superimposed image GB2 is obtained by applying transparency processing to the material image GB1 and superimposing the resulting image on the captured image GA4. FIG. 14 illustrates a display example of the superimposed image GB2 and a user interface image GA6. In the example shown in FIG. 14, the outlines of the multiple mountains, clouds, and sun in the material image GB1 are drawn with dotted lines, thereby expressing the translucency of the material image GB1. In the superimposed image GB2, the material image GB1 is translucent, allowing the user to view the captured image GA4 through the translucent material image GB1.

[0044] The second notification unit 210f outputs a notification urging the user to determine the position of the camera 30 so that the projection target SC in the captured image GA4 occupies the desired position in the material image GB1. In this embodiment, the second notification unit 210f displays a message M2 on the display device 420, such as "Move the camera so that the mannequin is in the desired position and press the 'OK' button," as shown in FIG. 14. In this embodiment, the notification is performed by displaying the message M2, but the notification may also be performed by outputting a sound representing the message M2. Upon viewing the message M2, the user moves with the camera 30 while checking the position of the projection target SC relative to the material image GB1 through the superimposed image GB2 displayed on the display device 420. As the camera 30 moves, the position of the projection target SC relative to the material image GB1 changes. Then, when the projection target SC occupies the desired position relative to the material image GB1, the user stops moving the camera 30 and performs a touch operation on the user interface image GA6. When the touch operation on the user interface image GA6 is performed, the measurement unit 210c performs the above-described 3D measurement.

[0045] The second generation unit 210g generates mask image data representing a mask image GB3 for extracting a region corresponding to the projection target SC from the material image GB1 based on one of a series of captured images captured during the course of performing 3D measurement. A specific example of the mask image GB3 is an image obtained by performing a transparency process or a process to make the region corresponding to the projection target SC in the captured image and painting the rest of the image black, as shown in FIG. 15. The second generation unit 210g overlays the mask image GB3 represented by the mask image data on the material image GB1 to obtain a masked material image GB4 in which the region in the material image GB1 other than the region corresponding to the projection target SC is painted black. FIG. 16 is a diagram showing an example of the masked material image GB4. The second generation unit 210g then generates projection image data by performing coordinate transformation indicated by the transformation data on the image data representing the masked material image GB4.

[0046] The projection image data generated by the second generation unit 210g is output to the projector 10 by the output unit 210e. The projector 10 projects the projection image represented by the projection image data output from the information processing device 20B onto the projection target SC. As a result, as shown in Fig. 17, a part of the material image GB1 is projected without distortion onto the surface of the projection target SC, just as if it were a pattern on a T-shirt.

[0047] Furthermore, the processing device 210 operating in accordance with the program PB executes the image generation method shown in Fig. 18. As shown in Fig. 18, the image generation method of this embodiment includes display control processing SA110, second notification processing SB120, measurement processing SA130, second generation processing SB140, and output processing SA150. The differences between the image generation method of this embodiment and the image generation method of the first embodiment are as follows:

[0048] In the second notification process SB120, the processing device 210 functions as a second notification unit 210f. In the second notification process SB120, the processing device 210 displays a message M2 on the display device 420, prompting the user to determine the position of the camera 30 so that the projection target SC shown in the captured image occupies a desired position in the material image GB1.

[0049] In the second generation process SB140, the processing device 210 functions as a second generation unit 210g. In the second generation process SB140, the processing device 210 generates a mask image GB3 based on one of a series of captured images captured in the course of executing the measurement process SA130. Next, the processing device 210 overlays the mask image GB3 on the material image GB1 to generate a masked material image GB4 in which parts of the material image GB1 other than the area corresponding to the projection target SC are painted black. Then, the processing device 210 generates projection image data by performing coordinate transformation indicated by the transformation data generated in the measurement process SA130 on the image data representing the masked material image GB4.

[0050] The projection image data generated in the second generation process SB140 is output in the output process SA150 to the projector 10. When the projection image represented by this projection image data is projected from the projector 10 onto the projection target SC, as shown in Fig. 17, a part of the material image GB1 is projected without distortion onto the surface of the projection target SC as if it were a pattern on a T-shirt.

[0051] As described above, this embodiment allows retail store staff to easily display products using projection mapping, even if they do not have specialized knowledge of projection mapping. Furthermore, this embodiment also eliminates the need to prepare product samples for each color and pattern of the product, thereby reducing waste of resources.

[0052] 3. Transformation The above embodiments can be modified as follows. (1) In the above embodiments, an example application of the present disclosure to projection mapping for displaying clothing products has been described. However, the present disclosure may also be applied to projection mapping for displaying products other than clothing, or to projection mapping for creating effects at theme parks, event venues, etc. By applying the present disclosure, it becomes possible for users who do not have specialized knowledge of projection mapping to create effects at theme parks, event venues, etc.

[0053] (2) In the above embodiments, one projector 10 projects an image onto one projection target SC. However, multiple projectors 10, each located at a different position, may project an image onto one projection target SC. Projecting images from multiple projectors 10, each located at a different position, onto one projection target SC can achieve projection mapping with increased brightness. Furthermore, projecting images from multiple projectors 10, each located at a different position, onto one projection target SC can minimize shadows and achieve projection mapping that can be viewed from any angle of 360°.

[0054] (3) The first notification processing SA120 in the first embodiment may be omitted. In an aspect in which the first notification processing SA120 is omitted, the first notification unit 210b may be omitted. This is because even if the first notification processing SA120 is omitted, the projection image can still be created accurately and easily. Similarly, the second notification processing SB120 in the second embodiment may also be omitted, and the second notification unit 210f may also be omitted. Furthermore, if the conversion data can be obtained separately, the measurement processing SA130 and the measurement unit 210c may also be omitted.

[0055] (4) The information processing device 20A may have a storage control unit that stores the projection image data generated by the first generation unit 210d in a storage device. This aspect makes it possible to reuse the projection image data. Specific examples of storage devices that store the projection image data include the storage device 230 of the information processing device 20A, the material management device 60, or a hard disk device that the processing device 210 can access through communication via the communication network 50. Similarly, the information processing device 20B may have a storage control unit that stores the projection image data generated by the second generation unit 210g in a storage device.

[0056] In an aspect in which projection image data is stored in a storage device, moving image data is generated in which projection images represented by each projection image data item are arranged in the time axis direction based on the plurality of projection image data items stored in the storage device, and each projection image is projected sequentially over time by the projector 10 in accordance with the moving image data items. Furthermore, data representing a new projection image obtained by arranging the projection images represented by each of the plurality of projection image data items in parallel or by overlapping them may be generated based on the plurality of projection image data items.

[0057] (5) In each of the above embodiments, the camera 30 and the terminal device 40 are separate devices. However, the camera 30 may be included in the terminal device 40. For example, if the terminal device 40 is a smartphone equipped with a camera, the camera of the smartphone may function as the camera 30. Furthermore, in the first embodiment, the information processing device 20A is a device different from the terminal device 40, the camera 30, and the projector 10. However, the information processing device 20A may be included in any of the terminal device 40, the camera 30, and the projector 10. Similarly, the information processing device 20B may be included in any of the terminal device 40, the camera 30, and the projector 10. In short, the image generation system of the present disclosure may include a display device and a processing device that executes the display control process SA110, either the first generation process SA140 or the second generation process SB140, and the output process SA150.

[0058] (6) In the first embodiment, the display control unit 210a, the first notification unit 210b, the measurement unit 210c, the first generation unit 210d, and the output unit 210e were software modules. However, any one or more, or all of the display control unit 210a, the first notification unit 210b, the measurement unit 210c, the first generation unit 210d, and the output unit 210e may be hardware modules such as an ASIC (Application Specific Integrated Circuit). Even if any one or more, or all of the display control unit 210a, the first notification unit 210b, the measurement unit 210c, the first generation unit 210d, and the output unit 210e are hardware modules, the same effects as those of the first embodiment can be achieved. Similarly, in the second embodiment, any one or more, or all of the display control unit 210a, the second notification unit 210f, the measurement unit 210c, the second generation unit 210g, and the output unit 210e may be hardware modules.

[0059] (7) The program PA may be manufactured as a standalone program or provided free of charge or for a fee. Specific examples of providing the program PA include providing the program PA by writing it to a computer-readable recording medium such as a flash ROM, or providing the program PA by downloading it via a telecommunications line such as the Internet. By operating a general computer in accordance with the program PA provided in these ways, it becomes possible to cause the computer to execute the image generation method of the present disclosure. Similarly, the program PB may be manufactured as a standalone program or provided free of charge or for a fee.

[0060] (8) In the above embodiments, the identification information D2 is stored in the storage device 230. However, a printed matter on which a two-dimensional barcode corresponding to the identification information D2 is printed may be attached to the housing of the information processing device 20A, the information processing device 20B, or the projector 10, and the terminal device 40 may acquire the identification information D2 by reading the two-dimensional barcode from the printed matter.

[0061] 4. Aspects grasped from at least one of each embodiment and each modified example The present disclosure is not limited to the above-described embodiments and modifications, and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following forms. The technical features in the above embodiments corresponding to the technical features in each form described below can be appropriately replaced or combined to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0062] One aspect of the image generation method of the present disclosure includes a display control process, a generation process, and an output process. The display process is a process of displaying a superimposed image in which a first image obtained by performing transparency processing on a first captured image is superimposed, and a user interface image. The first captured image is obtained by capturing an image of the projection object with a camera in a real space in which the projector and the projection object onto which the image from the projector is projected are located. The user interface image is an image for receiving an input for determining the position of the camera that captures the first captured image in the real space in which the projector and the projection object onto which the image from the projector is projected are located. The generation process is a process of generating a second image by correcting the first image according to the measurement results of the shape of the projection object. The first generation process SA140 in the first embodiment and the second generation process SB140 in the second embodiment are both aspects of the generation process of the present disclosure. In the generation process, the shape of the projection object may be measured based on a second captured image obtained by capturing an image of the projection object, onto which a pattern image for measuring the shape is projected from the projector, with the camera from a position determined by input to the user interface image. The output process is a process of outputting image data representing the second image to the projector. According to the image generation method of this aspect, even a user without specialized knowledge can easily perform projection mapping.

[0063] A more preferred embodiment of the image generating method may include a notification process that outputs a notification urging the user to determine a position where the angle of view of the first image and the angle of view of the first captured image match as the camera position. The first notification process SA120 in the first embodiment is one embodiment of the notification process in the present disclosure. According to this embodiment, the user can be urged to match the angle of view of the first image and the angle of view of the first captured image.

[0064] In a more preferred embodiment of the image generating method, the generation process may include generating a mask image based on the second captured image to extract a region corresponding to the projection object from the first image, and overlaying the mask image on the first image to generate a first image in which regions other than the region corresponding to the projection object are masked. In this embodiment, generating a second image by correcting the first image means generating the second image by correcting the first image in which regions other than the region corresponding to the projection object are masked, according to the shape of the projection object. This embodiment makes it possible to generate a second image by extracting a region corresponding to the projection object from the first image.

[0065] In a more preferred embodiment of the image generating method, when an input for determining the position of the camera is received, a signal for instructing the projector to project the pattern image can be output. According to this embodiment, the input for determining the position of the camera can be used as a trigger to start three-dimensional measurement of the projection target.

[0066] In a more preferred embodiment, the image generating method may further include a storage process for storing image data representing the second image in a storage device. This embodiment According to the present invention, a new projection image can be created using image data stored in a storage device.

[0067] Moreover, one aspect of the image generating system of the present disclosure includes a display device and a processing device that controls the display device. controlAccording to the image generation system of this aspect, even a user without specialized knowledge can easily perform projection mapping.

[0068] Furthermore, one aspect of the program of the present disclosure is to cause a computer to control According to the program of this aspect, even a user without specialized knowledge can easily perform projection mapping. [Explanation of symbols]

[0069] 1A, 1B...image generation system, 10...projector, 20A, 20B...information processing device, 30...camera, 40...terminal device, 50...communication line, 60...material management device, 210...processing device, 210a...display control unit, 210b...first acquisition unit, 210c...measurement unit, 210d...first generation unit, 210e...output unit, 210f...second notification unit, 210g...second generation unit, 220, 410...external IF device, 230...storage device, 420...display device, 430...input device, PA, PB...program, SC...projection object.

Claims

1. displaying a superimposed image in which a first image that has been subjected to transparency processing is superimposed on a first captured image obtained by capturing an image of a projection object by a camera in a real space in which a projector and a projection object onto which an image from the projector is projected are disposed, and a user interface image that receives an input for determining a position of the camera in the real space; generating a second image by correcting the first image in accordance with the shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector by the camera from the position; outputting image data representing the second image to the projector; and outputting a notification prompting the user to move the camera to a position where the angle of view of the first image and the angle of view of the first captured image match; An image generation method comprising:

2. generating the second image generating a mask image for extracting a region corresponding to the projection object from the first image based on the second captured image; and generating the first image in which an area other than an area corresponding to the projection object is masked using the mask image; Correcting the first image according to the shape of the projection object correcting the first image, in which an area other than an area corresponding to the projection object is masked, according to the shape of the projection object; The image generating method according to claim 1 .

3. 3. The image generating method according to claim 1, further comprising: when the input is received, outputting a signal instructing the projector to project the pattern image.

4. The image generating method of claim 1 , further comprising storing the image data in a storage device.

5. A display device; a processing device that controls the display device, The processing device includes: displaying, on the display device, a superimposed image in which a first image that has been subjected to transparency processing is superimposed on a first captured image obtained by capturing an image of a projection object with a camera in a real space in which a projector and a projection object onto which an image from the projector is projected are disposed, and a user interface image that receives an input for determining a position of the camera in the real space; generating a second image by correcting the first image in accordance with the shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector by the camera from the position; outputting image data representing the second image to the projector; and outputting a notification prompting the user to move the camera to a position where the angle of view of the first image and the angle of view of the first captured image match; Image generation system.

6. On the computer, displaying, on a display device, a superimposed image in which a first image that has been subjected to transparency processing is superimposed on a first captured image obtained by capturing an image of a projection object by a camera in a real space in which a projector and a projection object onto which an image from the projector is projected are disposed, and a user interface image that receives an input for determining a position of the camera in the real space; generating a second image by correcting the first image in accordance with the shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector by the camera from the position; outputting image data representing the second image to the projector; and outputting a notification prompting the user to move the camera to a position where the angle of view of the first image and the angle of view of the first captured image match; A program that executes the following.

Images