Image generation method, image generation system, and program

Abstract

To enable a user not having expertise to easily perform projection mapping.SOLUTION: An information processing apparatus 20A causes a display device 420 to display a superimposed image obtained by superimposing a material image applied with transmission processing on a captured image obtained by capturing a projection target object SC, which is a projection target of an image from a projector 10, with a camera 30, and a user interface image for receiving an input for determining a position of the camera 30. The information processing apparatus 20A generates projection image data representing a projection image by correcting the material image according to a three-dimensional shape measured on the basis of the captured image obtained by imaging, from the determined position, with the camera 30, the projection target object SC onto which a pattern image for three-dimensional measurement is projected from the projector 10, and outputs the projection image data to the projector 10.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image generation method, an image generation system, and a program.

Background Art

[0002] In recent years, projection mapping that performs various effects by projecting various images from a projector onto an object having a three-dimensional shape has been becoming popular. Hereinafter, an object onto which an image is projected from a projector is referred to as a projection target. To perform projection mapping, preliminary preparations such as distorting the image projected from the projector according to the three-dimensional shape of the projection target are required. This is because distortion corresponding to the three-dimensional shape of the projection target appears in the image reflected on the surface of the projection target. Various techniques for assisting the preliminary preparations in projection mapping have been proposed, and as an example thereof, the technique disclosed in Patent Document 1 can be mentioned. In the technique disclosed in Patent Document 1, a correction image including markers for position detection is projected onto a projection target by a projector. The correction image reflected on the surface of the projection target is imaged by a detection device, and markers are detected from the imaged correction image. By correcting the base image data based on the detection result of the markers, correction corresponding to the shape of the projection target is applied to the image projected from the projector.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to appropriately impart distortion to the image projected from a projector according to the three-dimensional shape of the object being projected, it is necessary that the markers are captured in the correction image without excess or deficiency, and with accurate shapes. If the markers are not captured in the correction image without excess or deficiency, and with accurate shapes, it will be difficult to detect the markers. In other words, the technology disclosed in Patent Document 1 assumes that the user has a certain level of expertise regarding projection mapping, such as expertise regarding imaging conditions such as the position and orientation of the detection device that captures the correction image. However, users who intend to perform projection mapping do not always possess expertise regarding projection mapping. The technology disclosed in Patent Document 1 has the problem that users without expertise in projection mapping cannot easily implement projection mapping. [Means for solving the problem]

[0005] One aspect of the image generation method of the present disclosure includes displaying a superimposed image obtained by superimposing a first image, which has undergone transparency processing, onto a first image obtained by imaging the projection target object with a camera in a real space where a projector and the projection target object are arranged, and a user interface image that accepts input for determining the position of the camera in the real space; generating a second image by correcting the first image according to the shape of the projection target object measured based on a second image obtained by imaging the projection target object being projected from the projector by the camera from the position, and outputting image data representing the second image to the projector.

[0006] Furthermore, one embodiment of the image generation system of the present disclosure includes a display device and a processing device for controlling the display device, wherein the processing device displays on the display device a superimposed image obtained by superimposing a first image obtained by imaging the projection target object with a camera in a real space where a projector and the projection target object to which an image from the projector is to be projected is arranged, a first image obtained by superimposing a first image that has undergone transparency processing, and a user interface image that accepts input for determining the position of the camera in the real space; generates a second image by correcting the first image according to the shape of the projection target object measured based on a second image obtained by imaging the projection target object projected from the projector by the camera from the position, and outputs image data representing the second image to the projector.

[0007] Furthermore, the program of this disclosure causes a computer to display on a display device a superimposed image obtained by superimposing a first image, which has undergone transparency processing, onto a first image obtained by superimposing a first image obtained by superimposing a first image obtained by superimposing a first image obtained by superimposing a first image obtained by superimposing a first image obtained by superimposing a first image obtained by superimposing a pattern image of the projection target being projected from the projector onto the projection target from the position using the camera, thereby generating a second image, and outputting image data representing the second image to the projector. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of the image generation system 1A according to the first embodiment of this disclosure. [Figure 2] This figure shows an example of an image represented by material data D1 in the first embodiment. [Figure 3] This figure shows an example configuration of the information processing device 20A. [Figure 4] This figure shows an example of a superimposed image. [Figure 5] This is a diagram to explain superimposed images. [Figure 6] This figure shows an example of the display of the superimposed image and the user interface image in the first embodiment. [Figure 7] This diagram illustrates the field of view of the source image and the captured image. [Figure 8] This figure shows an example of a superimposed image when the field of view of the source image matches the field of view of the captured image. [Figure 9] This figure shows an example of projection mapping realized by the first embodiment. [Figure 10] This flowchart shows the flow of the image generation method executed by the processing unit 210 of the information processing device 20A according to the program PA. [Figure 11] This figure shows an example configuration of the image generation system 1B according to the second embodiment of this disclosure. [Figure 12] This figure shows an example of an image represented by material data D1 in the second embodiment. [Figure 13] This figure shows an example configuration of the information processing device 20B. [Figure 14] This figure shows an example of the display of the superimposed image and the user interface image in the second embodiment. [Figure 15] This figure shows an example of a mask image. [Figure 16] This figure shows an example of a projection image generated using a mask image. [Figure 17] This figure shows an example of projection mapping implemented in the second embodiment. [Figure 18] This flowchart shows the flow of the image generation method executed by the processing unit 210 of the information processing device 20B according to the program PB. [Modes for carrying out the invention]

[0009] The embodiments described below are subject to various technically preferable limitations. However, the embodiments of the present disclosure are not limited to the forms described below. 1. First Embodiment FIG. 1 is a diagram showing a configuration example of an image generation system 1A according to a first embodiment of the disclosure of the present invention. The image generation system 1A is an information processing system that generates image data representing a projection image to be projected from a projector 10 onto a projection object SC in projection mapping. Hereinafter, the image data representing the projection image is referred to as projection image data. As shown in FIG. 1, the image generation system 1A includes an information processing device 20A and a terminal device 40. In addition to the image generation system 1A, FIG. 1 shows a projector 10, a camera 30, a projection object SC, a communication network 50, and a material management device 60.

[0010] The projection object SC in the present embodiment is a mannequin simulating the upper body of a person wearing a white and plain T-shirt. The projection object SC and the projector 10 are installed, for example, in the sales floor of a retail store selling clothing. In the present embodiment, a projection image corresponding to the color and pattern of the T-shirt is projected from the projector 10 onto the projection object SC, and a simulated product display is realized by projection mapping. Hereinafter, the store where the projection object SC and the projector 10 are installed is referred to as an implementation store.

[0011] The information processing device 20A is, for example, a stick-type personal computer. The information processing device 20A includes a male connector compliant with a predetermined standard such as USB (Universal Serial Bus). The projector 10 includes a female connector corresponding to the male connector. By inserting the male connector of the information processing device 20A into the female connector of the projector 10, the information processing device 20A and the projector 10 are electrically connected. In addition, the information processing device 20A communicates with the camera 30 and the terminal device 40 wirelessly or by wire.

[0012] The camera 30 is a device for imaging the projection target SC. The camera 30 is installed in the store with its optical axis directed towards the projection target SC using, for example, a tripod or the like. The camera 30 performs imaging under the control of the information processing device 20A and outputs image data representing the captured image to the information processing device 20A. Hereinafter, the image data representing the captured image is referred to as captured image data.

[0013] The communication network 50 is a telecommunication line such as the Internet, for example. The material management device 60 is connected to the communication network 50. The material management device 60 is, for example, a data server. One or more material data D1 are stored in advance in the material management device 60. The material data D1 is image data representing an image that serves as the basis for the projection image projected from the projector 10 onto the projection target SC. Hereinafter, the image represented by the material data is referred to as a material image. The material image is an example of the first image in the present disclosure. FIG. 2 is a diagram showing an example of the material image GA1 in the present embodiment. As shown in FIG. 2, the material image GA1 in the present embodiment is an image of a T-shirt having a pattern. The material data D1 is created by a designer or the like in charge of the design of the T-shirt and uploaded to the material management device 60. The material data D1 uploaded to the material management device 60 can be downloaded to the terminal device 40 by communication via the communication network 50.

[0014] The terminal device 40 is a smartphone used by the user of the projector 10. The user of the projector 10 in the present embodiment is a store clerk working in the store. As shown in FIG. 1, the terminal device 40 includes an external IF device 410, a display device 420, and an input device 430.

[0015] The external IF device 410 communicates with the material management device 60 via the communication network 50 and includes a communication circuit that communicates with the information processing device 20A. IF stands for Interface. The display device 420 includes a liquid crystal display and a driving circuit for the liquid crystal display. The terminal device 40 displays various images on the display device 420 under the control of the information processing device 20A. The input device 430 is a transparent sheet-like pressure sensor provided to cover the display area of ​​the display device 420 and accepts user input operations. The terminal device 40 transmits input operation data indicating the user's input operation to the input device 430 to the information processing device 20A via the external IF device 410. In this way, the user's input operation is transmitted to the information processing device 20A.

[0016] As will be explained in more detail later, the information processing device 20A performs the following operations in response to input operations made to the terminal device 40: capturing images with the camera 30, generating projection image data based on the captured image data acquired from the camera 30 and the material data D1 downloaded to the terminal device 40, and outputting the generated projection image data to the projector 10. The projection image represented by the projection image data generated by the information processing device 20A is projected from the projector 10 onto the projection target object SC, thereby realizing a product display using projection mapping.

[0017] Figure 3 shows an example configuration of the information processing device 20A. As shown in Figure 3, the information processing device 20A includes a processing unit 210, an external IF device 220, and a storage device 230. The processing unit 210 is composed of a processor, i.e., a computer, such as a CPU (Central Processing Unit). The processing unit 210 may consist of a single processor or multiple processors. The processing unit 210 functions as the control center of the information processing device 20A by operating according to the program PA stored in the storage device 230.

[0018] The external IF device 220 includes the aforementioned male connector. When this male connector is plugged into the female connector of the projector 10, and the information processing device 20A and the projector 10 are electrically connected, the external IF device 220 outputs data or signals provided by the processing device 210 to the projector 10. The external IF device 220 also includes a communication circuit for communicating with the camera 30 or the terminal device 40.

[0019] The storage device 230 is a recording medium that can be read by the processing unit 210. The storage device 230 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), or EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory).

[0020] The non-volatile memory of the storage device 230 pre-stores a program PA that causes the processing device 210 to execute the image generation method of the present disclosure. The non-volatile memory of the storage device 230 also pre-stores identification information D2 that uniquely identifies the material management device 60 within the communication network 50. A specific example of the identification information D2 is the communication address assigned to the material management device 60. The volatile memory of the storage device 230 is used by the processing device 210 as a work area when executing the program PA.

[0021] When the processing unit 210 detects a connection between the information processing unit 20A and the projector 10, it reads program PA from non-volatile memory to volatile memory and starts executing the read program PA. When the processing unit 210, which is operating according to program PA, detects a connection between the terminal device 40 and the camera 30 to the external IF device 410, it transmits identification information D2 to the terminal device 40. As a result, the terminal device 40 acquires the identification information D2.

[0022] When identification information D2 is acquired by the terminal device 40, the user uses the identification information D2 to access the material management device 60 and acquires material data D1, which represents the desired material image, by downloading it from the material management device 60 to the terminal device 40. When the processing device 210, which is operating according to program PA, detects that the terminal device 40 has acquired material data D1, it functions as the display control unit 210a, first notification unit 210b, measurement unit 210c, first generation unit 210d, and output unit 210e shown in Figure 3. The display control unit 210a, first notification unit 210b, measurement unit 210c, first generation unit 210d, and output unit 210e shown in Figure 3 are software modules realized by operating the processing device 210 according to program PA. The functions of each of the display control unit 210a, first notification unit 210b, measurement unit 210c, first generation unit 210d, and output unit 210e are as follows.

[0023] The display control unit 210a causes the camera 30 to take images at a predetermined period, such as every 1 millisecond. In this embodiment, the camera 30 is installed in the store with its optical axis facing the projection target SC, so the camera 30 takes images of the projection target SC. The image of the projection target SC captured by the camera 30 under the control of the display control unit 210a is an example of the first image in this disclosure. The display control unit 210a acquires image data from the camera 30 each time it takes an image. Each time image data is acquired, the display control unit 210a generates image data representing the superimposed image GA5 based on the material data D1 downloaded to the terminal device 40 and the acquired image data.

[0024] Figure 4 shows an example of the superimposed image GA5. Figure 5 is a diagram illustrating the superimposed image GA5. As shown in Figure 5, the superimposed image GA5 is generated by superimposing image GA3, obtained by applying transparency processing to the source image GA1 represented by source data D1, onto the captured image GA4 represented by the captured image data. Transparency processing is a process that sets the transmittance of the source image GA1 to a value greater than 0% and less than 100%. The closer the transmittance is to 0%, the less light is transmitted, and the closer it is to 100%, the more light is transmitted. The source image GA1 that has undergone transparency processing becomes semi-transparent. In the example shown in Figure 4, the semi-transparency of the source image GA1 is represented by drawing the outline of the T-shirt and the outline of the pattern applied to the T-shirt with dotted lines. In the superimposed image GA5, since the source image GA1 is semi-transparent, the user can view the captured image GA4 through the semi-transparent source image GA1. The transmittance in the transparency processing should be such that the captured image GA4 is visible through the source image GA1, and may be in the range of, for example, 10% to 90%. The user may be able to adjust the transmittance using a user interface (not shown).

[0025] The display control unit 210a provides the generated image data to the terminal device 40, causing the superimposed image GA5 to be displayed on the display device 420. The display control unit 210a also displays a user interface image GA6, which accepts input operations to determine the position of the camera 30 in the store, i.e., the position of the camera 30 in real space, on the display device 420 along with the superimposed image GA5. Figure 6 shows an example of the display of the superimposed image GA5 and the user interface image GA6. In the example shown in Figure 6, the user interface image GA6 is an image of a virtual operator that accepts touch operations from the user.

[0026] The first notification unit 210b outputs a notification prompting the user to determine the position of the camera 30 at a location where the field of view of the source image GA1 in the superimposed image GA5 matches the field of view of the captured image GA4 in the superimposed image GA5. In this embodiment, the first notification unit 210b displays the message M1 "Move the camera to a position where the mannequin and the product overlap and press the "OK" button" on the display device 420, as shown in Figure 6. In this embodiment, notification is given by displaying message M1, but notification may also be given by outputting an audio representing message M1.

[0027] Figure 7 shows an example of the field of view of the source image GA1 and the captured image GA4 in the superimposed image GA5. The field of view of the source image GA1 refers to the field of view of the camera when the source image GA1 was captured. The source image GA1 may be an image of an actual garment, or it may be an image created from that. If the source image GA1 is not an image of an actual garment, the field of view of the virtual camera that represents how the garment appears in the source image GA1 corresponds to the field of view of the source image GA1. The field of view of the captured image GA4 refers to the field of view of the camera 30 when the captured image GA4 was captured. The field of view is determined according to the position of the camera and the orientation of the optical axis. In Figure 7, the field of view of the source image GA1 is drawn with a dotted line, and the field of view of the captured image GA4 is drawn with a dashed line. In the example shown in Figure 7, the position of camera 30 when the captured image GA4 is captured is position P1. When camera 30 moves to position P2, the field of view of the source image GA1 and the field of view of the captured image GA4 approximately coincide. That is, in the example shown in Figure 7, position P2 is the position of camera 30 where the field of view of the source image GA1 in the superimposed image GA5 matches the field of view of the captured image GA4 in the superimposed image GA5.

[0028] Upon seeing message M1, the user moves the camera 30 while confirming the field of view of the source image GA1 and the field of view of the captured image G4 through the superimposed image GA5 displayed on the display device 420 of the terminal device 40. As the camera 30 moves, the field of view of the captured image GA4 included in the superimposed image GA5 displayed on the display device 420 changes. For example, when the camera 30 moves to position P2, the superimposed image GA5 shown in Figure 8 is displayed on the display device 420. When the field of view of the source image GA1 and the field of view of the captured image GA4 match, the user stops moving the camera 30 and performs a touch operation on the user interface image GA6.

[0029] When the measurement unit 210c receives a touch operation on the user interface image GA6, it performs a 3D measurement to measure the shape of the projection target object SC. More specifically, the measurement unit 210c outputs a signal to the projector 10 instructing it to project a series of pattern images for measuring the 3D shape of the object. Specific examples of pattern images include spatially encoding pattern images such as images representing Gray code patterns, or pattern images representing sine wave patterns. The measurement unit 210c also outputs a signal to the camera 30 to instruct it to take images, synchronized with the output of the signal instructing the projection of each pattern image, that is, delayed by a predetermined time from the output of the signal instructing the projection of the pattern image. As a result, the projection target object SC, onto which the series of pattern images are projected from the projector 10, is imaged for each pattern image by the camera 30 positioned at the location determined by the touch operation. In the example shown in Figure 7, the touch operation is performed at position P2, so the camera 30 positioned at position P2 takes images.

[0030] The measurement unit 210c generates conversion data for mutual coordinate transformation between the camera coordinate system and the world coordinate system based on a series of captured images and a series of pattern images obtained by capturing each pattern image of the projection target object SC, which is projected from the projector 10, from a position determined by the aforementioned touch operation using the camera 30. In other words, the measurement unit 210c measures the shape of the projection target object SC based on the captured images obtained by capturing the projection target object SC, which is projected from the projector 10, with the pattern images. Generating conversion data corresponds to measuring the shape of the projection target object SC. The series of captured images obtained by capturing each pattern image of the projection target object SC, which is projected from the projector 10, from a position determined by the touch operation using the camera 30 is an example of the second captured image in this disclosure. The camera coordinate system is a two-dimensional coordinate system that defines the position of the camera within the captured image. The world coordinate system is a three-dimensional coordinate system that defines the position in real space. Regarding the specific algorithm for generating conversion data from a series of pattern images, existing algorithms can be used as appropriate depending on the type of pattern image.

[0031] The first generation unit 210d generates projection image data by applying the coordinate transformation indicated by the transformation data to the material image GA1 represented by the material data D1. Applying the coordinate transformation indicated by the transformation data to the material image GA1 is equivalent to correcting the material image GA1 according to the shape of the projection target object SC. The output unit 210e then outputs the projection image data to the projector 10. The projector 10 projects the projection image represented by the projection image data output from the information processing device 20A onto the projection target object SC. The projection image is an example of the second image in this disclosure.

[0032] In this embodiment, the camera 30 is positioned so that the field of view of the captured image GA4 matches the field of view of the source image GA1. The fact that the field of view of the captured image GA4 matches the field of view of the source image GA1 means that the position of the camera relative to the subject when the source image GA1 is captured is approximately the same as the position of the camera 30 relative to the projection target SC, and that the direction of the optical axis of the camera when the source image GA1 is captured is approximately the same as the direction of the optical axis of the camera 30. Therefore, the fact that the field of view of the captured image GA4 matches the field of view of the source image GA1 means that the camera coordinate system for camera 30 and the camera coordinate system of the camera that captured the source image are approximately the same. Since the camera coordinate system for camera 30 is converted to the world coordinate system based on the conversion data, the camera coordinate system of the camera that captured the source image GA1 is also converted to the world coordinate system based on the conversion data. Since the converted source image GA1 is given distortion according to the shape of the projection target SC, as shown in Figure 9, the source image GA1 is projected onto the surface of the projection target SC with almost no distortion.

[0033] Furthermore, the processing unit 210, which operates according to program PA, executes the image generation method shown in Figure 10. As shown in Figure 10, the image generation method of this embodiment includes a display control process SA110, a first notification process SA120, a measurement process SA130, a first generation process SA140, and an output process SA150.

[0034] In the display control process SA110, the processing unit 210 functions as a display control unit 210a. In the display control process SA110, the processing unit 210 displays the superimposed image GA5 on the display device 420 based on the source data D1 downloaded to the terminal device 40 and the image data acquired from the camera 30, and also displays the user interface image GA6 on the display device 420.

[0035] In the first notification process SA120, the processing unit 210 functions as the first notification unit 210b. In the first notification process SA120, the processing unit 210 displays a message M1 on the display device 420 prompting the camera 30 to determine the position where the field of view of the source image GA1 matches the field of view of the captured image GA4.

[0036] In the measurement process SA130, the processing unit 210 functions as the measurement unit 210c. In the measurement process SA130, the processing unit 210 performs 3D measurement to measure the shape of the projection target SC and generates transformation data that mutually transforms the camera coordinate system of the camera 30 and the world coordinate system.

[0037] In the first generation process SA140, the processing unit 210 functions as the first generation unit 210d. In the first generation process SA140, the processing unit 210 generates projected image data by applying the coordinate transformation indicated by the transformation data generated in the measurement process SA130 to the material data D1 downloaded to the terminal device 40.

[0038] In output processing SA150, the processing unit 210 functions as an output unit 210e. In output processing SA150, the processing unit 210 outputs the projection image data generated in the first generation processing SA140 to the projector 10. The projector 10 projects the projection image represented by the projection image data output from the information processing unit 20A onto the projection target object SC. As the image represented by the projection image data is projected from the projector 10 onto the projection target object SC, the source image GA1 is projected onto the surface of the projection target object SC without distortion, as shown in Figure 9.

[0039] It is noteworthy that the user of the projector 10 is required to download the material data D1 and determine the position of the camera 30 in response to message M1 and perform a touch operation; no expertise in projection mapping is required. As described above, according to this embodiment, even if a retail store employee does not have expertise in projection mapping, it becomes possible to easily perform product displays using projection mapping.

[0040] In addition, when product displays for clothing are implemented using projection mapping, it becomes unnecessary to prepare product samples for each color and pattern of the product for the product display. Traditionally, retailers have changed their product lineup according to seasonal changes, and after the lineup has changed, the previous product samples are no longer needed. These unwanted product samples were sometimes sold at a low price, but most were discarded. Such discarding of product samples was a problem from the perspective of efficient use of resources. According to this embodiment, the waste of resources is reduced.

[0041] 2. Second Embodiment Figure 11 shows an example of the configuration of an image generation system 1B according to a second embodiment of the present disclosure. In Figure 11, the same components as in Figure 1 are denoted by the same reference numerals. In addition to the image generation system 1B, Figure 11 also shows a projection target SC, a projector 10, a camera 30, a communication network 50, and a material management device 60, similar to Figure 1. As is clear from comparing Figure 11 with Figure 1, the configuration of the image generation system 1B differs from that of the image generation system 1A in that it has an information processing device 20B instead of an information processing device 20A. That is, the image generation system 1B includes an information processing device 20B and a terminal device 40. In this embodiment, the material data D1 stored in the material management device 60 is image data representing an image of a pattern such as fabric or wallpaper, an image of an animal or a person, or a landscape image. In this embodiment, the material image GB1 represented by the material data D1 is a landscape image showing multiple mountains, clouds floating in the sky, and the sun, as shown in Figure 12.

[0042] The information processing device 20B is a stick-type personal computer, similar to the information processing device 20A. Figure 13 shows an example of the configuration of the information processing device 20B. In Figure 13, the same components as in Figure 3 are denoted by the same reference numerals. As is clear from comparing Figure 13 and Figure 3, the hardware configuration of the information processing device 20B is the same as that of the information processing device 20A. That is, the information processing device 20B has a processing device 210, an external IF device 220, and a storage device 230. The configuration of the information processing device 20B differs from that of the information processing device 20A in that program PB is stored in the storage device 230 instead of program PA. The processing device 210, which operates according to program PB, functions as the display control unit 210a, second notification unit 210f, measurement unit 210c, second generation unit 210g, and output unit 210e shown in Figure 13.

[0043] The difference between the configuration of the information processing device 20B and the information processing device 20A is that a second notification unit 210f is provided instead of the first notification unit 210b, and a second generation unit 210g is provided instead of the first generation unit 210d. However, in this embodiment, since the source image GB1 is a landscape image, the superimposed image GB2 that the display control unit 210a displays on the display device 420 is also different from the superimposed image GA5 in the first embodiment. The superimposed image GB2 is an image obtained by superimposing an image obtained by applying transparency processing to the source image GB1 onto the captured image GA4. Figure 14 shows an example of the display of the superimposed image GB2 and the user interface image GA6. In the example shown in Figure 14, the fact that the source image GB1 is semi-transparent is expressed by drawing the outlines of each of the mountains, clouds, and sun reflected in the source image GB1 with dotted lines. In the superimposed image GB2, since the source image GB1 is semi-transparent, the user can view the captured image GA4 through the semi-transparent source image GB1.

[0044] The second notification unit 210f outputs a notification prompting the user to determine the position of the camera 30 so that the projection target object SC, which is captured in the image GA4, occupies a desired position in the source image GB1. In this embodiment, the second notification unit 210f displays the message M2, "Move the camera so that the mannequin is in your preferred position and press the "OK" button," on the display device 420, as shown in Figure 14. In this embodiment, notification is given by displaying message M2, but notification may also be given by outputting an audio representation of message M2. Upon seeing message M2, the user moves the camera 30 while confirming the position of the projection target object SC relative to the source image GB1 through the superimposed image GB2 displayed on the display device 420. As the camera 30 moves, the position of the projection target object SC relative to the source image GB1 changes. Then, when the projection target object SC occupies the desired position relative to the source image GB1, the user stops moving the camera 30 and performs a touch operation on the user interface image GA6. When a touch operation is performed on the user interface image GA6, the measurement unit 210c performs the aforementioned 3D measurement.

[0045] The second generation unit 210g generates mask image data representing a mask image GB3 for extracting the region corresponding to the projection target SC from the source image GB1, based on any of the series of captured images taken during the process of performing 3D measurement. A specific example of the mask image GB3 is an image obtained by applying a transparency process or making the region corresponding to the projection target SC transparent in the captured image, and filling in the parts other than that region with black, as shown in Figure 15. The second generation unit 210g obtains a masked source image GB4 by superimposing the mask image GB3 represented by the mask image data onto the source image GB1, thereby filling in the parts of the source image GB1 other than the region corresponding to the projection target SC with black. Figure 16 shows an example of the masked source image GB4. The second generation unit 210g then generates projection image data by applying the coordinate transformation indicated by the transformation data to the image data representing the masked source image GB4.

[0046] The projection image data generated by the second generation unit 210g is output to the projector 10 by the output unit 210e. The projector 10 projects the projection image represented by the projection image data output from the information processing device 20B onto the projection target object SC. As a result, as shown in Figure 17, a part of the source image GB1 is projected onto the surface of the projection target object SC without distortion, as if it were the pattern on a T-shirt.

[0047] Furthermore, the processing unit 210, which operates according to program PB, executes the image generation method shown in Figure 18. As shown in Figure 18, the image generation method of this embodiment includes a display control process SA110, a second notification process SB120, a measurement process SA130, a second generation process SB140, and an output process SA150. The differences between the image generation method of this embodiment and the image generation method of the first embodiment are as follows:

[0048] In the second notification process SB120, the processing unit 210 functions as a second notification unit 210f. In the second notification process SB120, the processing unit 210 causes the display device 420 to display a message M2 prompting the camera 30 to determine its position so that the projection target SC captured in the image occupies a desired position in the source image GB1.

[0049] In the second generation process SB140, the processing unit 210 functions as a second generation unit 210g. In the second generation process SB140, the processing unit 210 generates a mask image GB3 based on one of the series of captured images taken during the execution of the measurement process SA130. Next, the processing unit 210 generates a masked material image GB4 by superimposing the mask image GB3 onto the material image GB1, thereby blacking out the areas in the material image GB1 that do not correspond to the projection target SC. Then, the processing unit 210 generates projection image data by applying the coordinate transformation indicated by the transformation data generated in the measurement process SA130 to the image data representing the masked material image GB4.

[0050] The projection image data generated in the second generation process SB140 is output to the projector 10 in the output process SA150. When the projection image represented by this projection image data is projected from the projector 10 onto the projection target object SC, a part of the source image GB1 is projected onto the surface of the projection target object SC without distortion, as shown in Figure 17, as if it were the pattern on a T-shirt.

[0051] As described above, this embodiment makes it possible to easily create product displays using projection mapping even if retail store employees do not have specialized knowledge of projection mapping. Furthermore, this embodiment also eliminates the need to prepare product samples for each color and pattern of the product, thus reducing resource waste.

[0052] 3. Transformation Each of the above embodiments can be modified as follows. (1) In each of the above embodiments, examples of applying the present disclosure to projection mapping for realizing clothing product displays have been described. However, the present disclosure may also be applied to projection mapping for realizing product displays other than clothing, or to projection mapping for realizing performances in theme parks or event venues. By applying the present disclosure, it becomes possible for users without specialized knowledge of projection mapping to realize projection mapping for performances in theme parks or event venues.

[0053] (2) In each of the above embodiments, one projector 10 projected an image onto a single projection target SC. However, multiple projectors 10, each positioned at a different location, may project an image onto a single projection target SC. By projecting images from multiple projectors 10, each positioned at a different location, onto a single projection target SC, projection mapping with increased brightness can be realized. Furthermore, by projecting images from multiple projectors 10, each positioned at a different location, onto a single projection target SC, shadows can be minimized, and projection mapping that can be viewed from any angle (360°) can be realized.

[0054] (3) The first notification process SA120 in the first embodiment may be omitted. In the embodiment in which the first notification process SA120 is omitted, the first notification unit 210b may also be omitted. This is because even if the first notification process SA120 is omitted, the projected image can still be created accurately and easily. Similarly, the second notification process SB120 in the second embodiment can also be omitted, and the second notification unit 210f can also be omitted. Furthermore, if conversion data can be obtained separately, the measurement process SA130 and the measurement unit 210c can also be omitted.

[0055] (4) The information processing device 20A may have a storage control unit that stores the projection image data generated by the first generation unit 210d in a storage device. According to this embodiment, the projection image data can be reused. Specific examples of storage devices that serve as storage destinations for projection image data include the storage device 230 of the information processing device 20A, the material management device 60, or a hard disk drive accessible to the processing device 210 via communication over the communication network 50. Similarly, the information processing device 20B may have a storage control unit that stores the projection image data generated by the second generation unit 210g in a storage device.

[0056] In an embodiment in which projection image data is stored in a storage device, dynamic image data is generated by arranging the projection images represented by each projection image data in the time axis direction based on the multiple projection image data stored in the storage device, and each projection image can be sequentially projected onto the projector 10 as time progresses according to the dynamic image data. Alternatively, data representing a new projection image obtained by arranging or overlapping each projection image represented by each of the multiple projection image data may be generated based on the multiple projection image data.

[0057] (5) In each of the above embodiments, the camera 30 and the terminal device 40 were separate devices, but the camera 30 may be included in the terminal device 40. For example, if the terminal device 40 is a smartphone equipped with a camera, the camera of the smartphone may perform the role of the camera 30. Also, in the first embodiment, the information processing device 20A was a device different from the terminal device 40, the camera 30, and the projector 10. However, the information processing device 20A may be included in the terminal device 40, the camera 30, and the projector 10. Similarly, the information processing device 20B may be included in the terminal device 40, the camera 30, and the projector 10. In short, the image generation system of the present disclosure only needs to include a display device and a processing device that performs either the display control processing SA110, the first generation processing SA140, or the second generation processing SB140, and the output processing SA150.

[0058] (6) In the first embodiment, the display control unit 210a, the first notification unit 210b, the measurement unit 210c, the first generation unit 210d, and the output unit 210e were software modules. However, one or more of the display control unit 210a, the first notification unit 210b, the measurement unit 210c, the first generation unit 210d, and the output unit 210e, or all of them, may be hardware modules such as ASICs (Application Specific Integrated Circuits). Even if one or more of the display control unit 210a, the first notification unit 210b, the measurement unit 210c, the first generation unit 210d, and the output unit 210e, or all of them, are hardware modules, the same effects as in the first embodiment will be achieved. Similarly, in the second embodiment, one or more of the display control unit 210a, the second notification unit 210f, the measurement unit 210c, the second generation unit 210g, and the output unit 210e, or all of them, may be hardware modules.

[0059] (7) Program PA may be manufactured as a standalone unit and may be provided for a fee or free of charge. Specific methods of providing Program PA include writing Program PA to a computer-readable recording medium such as flash ROM and providing it, or providing Program PA by downloading it via a telecommunications line such as the Internet. By operating a general-purpose computer according to Program PA provided in these ways, it becomes possible to make the computer execute the image generation method of the present disclosure. Similarly, Program PB may be manufactured as a standalone unit and may be provided for a fee or free of charge.

[0060] (8) In each of the above embodiments, the identification information D2 was stored in the storage device 230. However, a method may be adopted in which a printed document with a two-dimensional barcode corresponding to the identification information D2 is attached to the housing of the information processing device 20A, the information processing device 20B, or the projector 10, and the terminal device 40 acquires the identification information D2 by reading the two-dimensional barcode from the printed document.

[0061] 4. Aspects that can be understood from at least one of each embodiment and each modified example. This disclosure is not limited to the embodiments and modifications described above, and can be implemented in various ways without departing from its spirit. For example, this disclosure can also be implemented in the following ways. The technical features in the embodiments described below that correspond to the technical features in each of the embodiments described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure or to achieve some or all of the effects of this disclosure. Furthermore, if the technical features are not described as essential in this specification, they can be deleted as appropriate.

[0062] One embodiment of the image generation method of the present disclosure includes a display control process, a generation process, and an output process. The display process is a process of displaying a superimposed image, obtained by superimposing a first image obtained by applying a transparency process to a first captured image, and a user interface image. The first captured image is obtained by imaging the projection target object with a camera in a real space where a projector and the projection target object are arranged. The user interface image is an image for receiving input to determine the position of the camera that captures the first captured image in the real space where the projector and the projection target object are arranged. The generation process is a process of generating a second image by correcting the first image according to the measurement result of the shape of the projection target object. The first generation process SA140 in the first embodiment and the second generation process SB140 in the second embodiment are both embodiments of the generation process of the present disclosure. In the generation process, the shape of the projection target object may be measured based on a second captured image obtained by imaging the projection target object, which is projected from a projector with a pattern image for shape measurement, from a position determined by the input to the user interface image. The output process involves outputting image data representing the second image to a projector. According to this image generation method, even users without specialized knowledge can easily perform projection mapping.

[0063] A more preferred embodiment of the image generation method may include a notification process that outputs a notification prompting the user to determine the camera position at a location where the field of view of the first image matches the field of view of the first captured image. The first notification process SA120 in the first embodiment is one embodiment of the notification process in this disclosure. According to this embodiment, the user can be prompted to match the field of view of the first image with the field of view of the first captured image.

[0064] In a more preferred embodiment of the image generation method, the generation process may include generating a mask image based on a second captured image for extracting a region corresponding to the projection target from a first image, and generating a first image in which regions other than those corresponding to the projection target are masked by superimposing the mask image onto the first image. Generating a second image by correcting the first image in this embodiment means generating a second image by correcting the first image, in which regions other than those corresponding to the projection target are masked, according to the shape of the projection target. According to this embodiment, it becomes possible to extract a region corresponding to the projection target from the first image and generate a second image.

[0065] A more preferred embodiment of the image generation method may further include outputting a signal to a projector instructing it to project a pattern image when it receives an input determining the camera position. According to this embodiment, three-dimensional measurement of the projection target can be initiated when an input determining the camera position is received.

[0066] A more preferred embodiment of the image generation method may further include a storage process for storing image data representing a second image in a storage device. According to this embodiment, a new projection image can be created using the image data stored in the storage device.

[0067] Furthermore, one embodiment of the image generation system of this disclosure includes a display device and a processing device for controlling the display device. The processing device performs the aforementioned display processing, generation processing, and output processing. According to the image generation system of this embodiment, even users without specialized knowledge can easily perform projection mapping.

[0068] Furthermore, one embodiment of the program of this disclosure causes a computer to perform the aforementioned display processing, generation processing, and output processing. According to the program of this embodiment, even users without specialized knowledge can easily perform projection mapping. [Explanation of Symbols]

[0069] 1A, 1B...Image generation system, 10...Projector, 20A, 20B...Information processing device, 30...Camera, 40...Terminal device, 50...Communication line, 60...Material management device, 210...Processing device, 210a...Display control unit, 210b...First acquisition unit, 210c...Measurement unit, 210d...First generation unit, 210e...Output unit, 210f...Second notification unit, 210g...Second generation unit, 220, 410...External IF device, 230...Storage device, 420...Display device, 430...Input device, PA, PB...Program, SC...Projection target.

Claims

1. displaying a superimposed image obtained by superimposing a first image that has been subjected to transparency processing on a first captured image obtained by capturing an image of a projection object by a camera in a real space in which a projector and a projection object onto which an image from the projector is projected are disposed, and a user interface image that accepts an input for determining a position of the camera in the real space; generating a second image by correcting the first image in accordance with a shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector by the camera from the position; and outputting image data representing the second image to the projector; 13. An image generating method comprising:

2. The image generating method according to claim 1 , further comprising: outputting a notification prompting the user to move the camera to a position where an angle of view of the first image and an angle of view of the first captured image match.

3. Producing the second image includes: generating a mask image for extracting a region corresponding to the projection object from the first image based on the second captured image; and generating the first image in which an area other than an area corresponding to the projection object is masked using the mask image; Correcting the first image in accordance with the shape of the projection object correcting the first image, in which an area other than an area corresponding to the projection object is masked, according to a shape of the projection object; The image generating method according to claim 1 .

4. The image generating method according to claim 1 , further comprising: when the input is received, outputting, to the projector, a signal instructing the projector to project the pattern image.

5. The method of claim 1 , further comprising storing the image data in a storage device.

6. A display device; A processing device that controls the display device, The processing device includes: displaying on the display device a superimposed image obtained by superimposing a first image that has been subjected to transparency processing on a first captured image obtained by capturing an image of a projection object by a camera in a real space in which a projector and a projection object onto which an image from the projector is projected are disposed, and a user interface image that receives an input for determining a position of the camera in the real space; generating a second image by correcting the first image in accordance with a shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector by the camera from the position; and outputting image data representing the second image to the projector. Image generation system.

7. On the computer, displaying, on a display device, a superimposed image obtained by superimposing a first image that has been subjected to transparency processing on a first captured image obtained by capturing an image of a projection object by a camera in a real space in which a projector and a projection object onto which an image from the projector is projected are disposed, and a user interface image that accepts an input for determining a position of the camera in the real space; generating a second image by correcting the first image in accordance with a shape of the projection object measured based on a second captured image obtained by capturing an image of the projection object onto which a pattern image is projected from the projector by the camera from the position; and outputting image data representing the second image to the projector; A program that executes the following.

Images