Display method

Abstract

To generate and display a content not intended by a user.SOLUTION: A video generator 20 includes a first acquisition section 211, a second acquisition section 212, a generation section 213, and a display control section 214. The first acquisition section 211 acquires first environment information representing an environment of a projection area in which a projection object, onto which a projector projects an image, is installed. The first acquisition section 211 also acquires second environment information representing an environment at a time later than a time point at which the first environment information is acquired. The second acquisition section 212 acquires a difference between the first environment information and the second environment information. The generation section 213 generates a projection image to be projected from the projector onto the projection object on the basis of the difference acquired by the second acquisition section 212. The display control section 214 displays the projection image generated by the generation section 213 in the projection area.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a display method. [Background technology]

[0002] Patent Document 1 discloses that even if the position of a seated person changes, the projected image can be automatically adapted in accordance with each change. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 116290 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology disclosed in Patent Document 1, the user creates content in advance, and the projector simply displays that content, which may increase the workload of the user creating the content. [Means for solving the problem]

[0005] One aspect of the display method of the present disclosure includes acquiring first environmental information representing the environment of a projection area onto which an image is projected by a projector, acquiring second environmental information representing the environment at a time later than the time the first environmental information was acquired, acquiring a difference between the first environmental information and the second environmental information, generating a projection image to be projected onto the projection area from the projector based on the difference, and displaying the projection image in the projection area by projecting the projection image onto the projection area from the projector. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a diagram illustrating an example configuration of a system 1 according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of projection mapping by the system 1. [Figure 3] 2 is a diagram showing an example of the configuration of a video generator 20 included in the system 1. FIG. [Figure 4] 10 is a flowchart showing the flow of processing in a display method executed by a processing device 210 of an image generator 20 in accordance with a program PRA. DETAILED DESCRIPTION OF THE INVENTION

[0007] The embodiments described below are subject to various technically preferable limitations, but the embodiments of the present disclosure are not limited to the following embodiments. 1. Embodiment Fig. 1 is a diagram illustrating an example configuration of a system 1 according to an embodiment of the present disclosure. The system 1 is a system that realizes projection mapping by projecting a projection image GA onto a projection target SC. As shown in Fig. 1, the system 1 includes a projector 10, an image generator 20, and a measuring instrument 30. The projector 10 and the measuring instrument 30 are each connected to the image generator 20 wirelessly or via a wired connection.

[0008] The projector 10 achieves projection mapping by projecting a projection image GA represented by image data supplied from the image generator 20 onto a projection area SR where a projection target SC is placed. In this embodiment, the projection area SR is, for example, the floor of a living room in a user's home, and the projection target SC is a ball placed in the projection area SR. The measuring instrument 30 is, for example, a video camera. The measuring instrument 30 is placed at a position and orientation such that the entire projection area SR fits within the imaging field of view. The measuring instrument 30 captures an image of the projection area SR under the control of the image generator 20. The measuring instrument 30 supplies image data representing the captured image obtained by capturing the projection area SR to the image generator 20.

[0009] The image generator 20 is, for example, a personal computer. Based on the captured image data supplied from the measuring instrument 30, the image generator 20 generates projection image data representing the projection image GA to be projected from the projector 10 onto the projection area SR. As will be described in detail later, in this embodiment, the image generator 20 detects any changes in the environment of the projection area SR based on the captured image data supplied from the measuring instrument 30 and generates projection image data corresponding to the detected changes. A specific example of a change in the environment of the projection area SR in this embodiment is the movement of the projection target SC within the projection area SR. Specific examples of the movement of the projection target SC include a change in the position occupied by the projection target SC within the projection area SR (i.e., movement of the projection target SC), vibration or rotation of the projection target SC without changing its position, and a combination of any two or all of these. In this embodiment, when the projection target SC moves, projection mapping is used to impart an effect, such as adding a polka dot pattern to the surface of the projection target SC, as shown in FIG. 2 . 2, the movement of the projection target SC is indicated by an arrow Y, and the projection target SC before movement is drawn with a dotted line. Also, a mode may be adopted in which at least one of the size and color of the pattern is changed according to the movement speed of the projection target SC. For example, a mode may be adopted in which red polka dots are applied to the projection target SC when the movement speed of the projection target SC is equal to or greater than 0 and less than a first threshold, and blue polka dots are applied when the movement speed of the projection target SC is equal to or greater than the first threshold and less than a second threshold that is greater than the first threshold.

[0010] In this embodiment, the application of effects to the projection target SC is realized by projection mapping, but the display of a story-like video that starts when the projection target SC is placed in the projection area SR may also be realized by projection mapping. For example, if the projection target SC is a plastic bottle or candy, placing the plastic bottle in the projection area SR could cause a caterpillar generated in the video to start walking out of the bottle, and if a candy is further placed in the projection area SR, an animation video could be output in which the caterpillar starts walking towards the candy and appears to be eating the candy.

[0011] FIG. 3 is a diagram showing an example of the configuration of the image generator 20. As shown in FIG. 3, the image generator 20 includes a processing device 210, a communication device 220, and a storage device 230. The processing device 210 is one or more processors. The processing device 210 is, for example, a CPU (Central Processing Unit). The processing device 210 functions as the control center of the image generator 20 by operating in accordance with a program PRA stored in the storage device 230. The communication device 220 is a device that performs wireless or wired communication with other devices, and includes, for example, an interface circuit. Specific examples of other devices that communicate with the communication device 220 include the projector 10 and a measuring instrument 30.

[0012] The storage device 230 is a recording medium readable by the processing device 210. The storage device 230 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Radom Access Memory). Various programs and a learning model file MDL are stored in the nonvolatile memory of the storage device 230.

[0013] The learning model file MDL is data corresponding to an inference model that has learned the correspondence between image generation conditions and images generated in accordance with those conditions. Well-known techniques may be appropriately employed for the learning model file MDL. In this embodiment, the generation conditions are changes in the environment of the projection area SR. In this embodiment, the learning model file MDL pre-associates, for example, generation conditions indicating the movement of the projection target SC with effect images that add polka dots to the projection target SC. Note that, in a mode that realizes the display of the story-like video described above, it is sufficient to pre-associate generation conditions indicating the type of projection target SC with animated images that show the movement of a caterpillar when that type of object is placed. In this embodiment, the processing device 210 functions as an image generation AI that uses the learning model file MDL to perform comprehensive data analysis of environmental changes in the projection area SR and generate images in response to those environmental changes.

[0014] Examples of various programs stored in the non-volatile memory include a kernel program and a program PRA. The kernel program is not shown in Figure 3. The kernel program is a program that causes the processing device 210 to implement an OS (Operating System).

[0015] When the image generator 20 is powered on, the processing device 210 reads the kernel program from the nonvolatile memory to the volatile memory and starts executing the read kernel program. When the processing device 210, which is operating according to the kernel program, receives an instruction to start executing another program via an input device (not shown), the processing device 210 starts executing the other program. For example, when an instruction to start executing the program PRA is received, the processing device 210 reads the program PRA from the nonvolatile memory to the volatile memory and starts executing the program PRA read into the volatile memory.

[0016] The processing device 210 operating in accordance with the program PRA functions as the first acquisition unit 211, the second acquisition unit 212, the generation unit 213, and the display control unit 214 shown in Fig. 3. In other words, each of the first acquisition unit 211, the second acquisition unit 212, the generation unit 213, and the display control unit 214 shown in Fig. 3 is a software module realized by operating the processing device 210 in accordance with the program PRA. The roles of each of the first acquisition unit 211, the second acquisition unit 212, the generation unit 213, and the display control unit 214 shown in Fig. 3 are as follows.

[0017] The first acquisition unit 211 starts up the measuring instrument 30 by executing a first startup process that starts up the measuring instrument 30. When the startup of the measuring instrument 30 is complete, the first acquisition unit 211 causes the measuring instrument 30 to capture an image of the projection area SR and acquires first captured image data that represents the first captured image that is the image capture result. If the projector 10 is not started up at the time the first captured image data is acquired, the first captured image does not include the projected image GA projected from the projector 10 onto the projection area SR. The first captured image is an example of first environmental information in the present disclosure.

[0018] Furthermore, the first acquisition unit 211 causes the measuring instrument 30 to capture an image of the projection area SR at a second time that is later than the first time at which the first captured image data is acquired, and acquires second captured image data that represents the second captured image that is the imaging result. If the projector 10 is not running at the time the second captured image data is acquired, the second captured image does not include the projected image GA projected onto the projection area SR from the projector 10. The second captured image is an example of second environment information in the present disclosure.

[0019] The second acquisition unit 212 acquires the difference between the first environmental information and the second environmental information, i.e., the difference between the first captured image data and the second captured image data. Specific examples of the difference include a change in the contour of the projection target SC, or a change in the position of a natural feature point on the surface of the projection target SC. Alternatively, a marker such as an AR marker may be attached to the projection target SC in advance, and the change in the position of the marker may be used as the difference.

[0020] In this embodiment, the second acquisition unit 212 identifies the projection target SC and detects the contour of the projection target SC by performing object recognition on each of the first captured image data and the second captured image data. The second acquisition unit 212 then acquires the change in the position of the contour of the projection target SC in the captured images represented by each of the first captured image data and the second captured image data as the difference. Note that for captured image data acquired after the projector 10 starts projecting the projection image GA, the second acquisition unit 212 may acquire the difference after performing image processing to remove the projection image GA. This is to prevent the projection of the projection image GA itself from being detected as a change in the environment. Focusing only on the changes in the environment reduces the computational load. Furthermore, if the change in the environment is related to the theme of the content, the user's workload can be reduced while content with an easily understandable theme can be provided.

[0021] The generation unit 213 generates content data representing a projection image GA to be projected from the projector 10 onto the projection area SR, based on the difference acquired by the second acquisition unit 212. In other words, the generation unit 213 recognizes the change portion from the environment represented by the first environmental information to the environment represented by the second environmental information, based on the difference, and inputs data representing the change portion to the image generation AI described above, thereby generating content data representing a projection image GA according to the change in the environment.

[0022] The display control unit 214 transmits the content data generated by the generation unit 213 to the projector 10, and displays the projection image GA represented by the content data in the projection area SR by having the projector 10 project the projection image GA. As a result, a projection image GA corresponding to a change in the environment in the projection area SR (in this embodiment, the movement of the projection target SC) is displayed in the projection area SR. Note that if the projector 10 is running when the content data generated by the generation unit 213 is generated, the display control unit 214 executes a second startup process to start up the projector 10, and then transmits the content data generated by the generation unit 213 to the projector 10.

[0023] Furthermore, the processing device 210, operating in accordance with the program PRA, executes a display method emphasizing features of the present disclosure. Fig. 4 is a flowchart showing the process flow of this display method. As shown in Fig. 4, this display method includes a first acquisition process SA110, a second acquisition process SA120, a generation process SA130, a display control process SA140, and a determination process SA150.

[0024] In the first acquisition process SA110, the processing device 210 functions as a first acquisition unit 211. In the first acquisition process SA110, the processing device 210 controls the measuring instrument 30 to acquire the first captured image data and the second captured image data.

[0025] In the second acquisition process SA120 subsequent to the first acquisition process SA110, the processing device 210 functions as the second acquisition section 212. In the second acquisition process SA120, the processing device 210 acquires the difference between the first captured image data and the second captured image data.

[0026] In the generation process SA130, the processing device 210 functions as a generation unit 213. In the generation process SA130, the processing device 210 generates content data representing a projection image GA to be projected from the projector 10 onto the projection target SC, based on the environmental change data representing the difference acquired in the second acquisition process SA120 and the learning model file MDL. In the display control process SA140 that follows the generation process SA130, the processing device 210 functions as a display control unit 214. In the display control process SA140, the processing device 210 transmits the content data generated in the generation process SA130 to the projector 10, and displays the projection image GA by having the projector 10 project an image represented by the content data. In this embodiment, the first acquisition process SA110 is executed prior to the display control process SA140, so if the measuring instrument 30 is not started and the projector 10 is not started at the start of this display method, the following steps are executed in this order: a first startup process to start the measuring instrument 30, after executing the first startup process, a first captured image is captured, after capturing the first captured image, a second captured image is captured, after executing the first startup process, a second startup process to start the projector 10, and after executing the second startup process, a projected image GA is displayed in the projection area SR.

[0027] In determination processing SA150 following display control processing SA140, the processing device 210 determines whether or not the user has instructed the device to end the display of the projected image GA. If the user has instructed the device to end the display of the projected image GA, the determination result of determination processing SA150 is "Yes." If the determination result of determination processing SA150 is "Yes," the processing device 210 ends the execution of this display method. In contrast, if the user has not instructed the device to end the display of the projected image GA, the determination result of determination processing SA150 is "No." If the determination result of determination processing SA150 is "No," the processing device 210 re-executes the first acquisition processing SA110 and subsequent processing.

[0028] As described above, according to this embodiment, a projection image GA corresponding to changes in the environment of the projection area SR is displayed in the projection area SR. This projection image GA is generated using image generation AI, and is therefore an image unrelated to the user's intention. Therefore, according to this embodiment, a projection image GA not intended by the user can be generated and displayed in the projection area SR, thereby attracting the user's interest. Furthermore, according to this embodiment, by utilizing the projector 10, the projection image GA can be viewed in the same space as the space to which the changes have been made. Furthermore, according to this embodiment, no information other than the changes in the environment of the projection area SR is required to generate the projection image GA, so even users without specialized knowledge of image generation can easily enjoy projection mapping.

[0029] 2. Other embodiments (1) In the above embodiment, the measuring instrument 30 is an imaging device. However, it may be a depth sensor that measures the relative positional relationship (e.g., the distance between) between the projection target SC, onto which the image is projected by the projector 10, and the projector 10. In this case, the first acquisition unit 211 acquires first relative position information that represents the relative positional relationship between the projection target SC and the projector 10 at a certain time. The first acquisition unit 211 also acquires second relative position information that represents the relative positional relationship between the projection target SC and the projector 10 at a second time that is later than the first time at which the first relative position information is acquired. The second acquisition unit 212 acquires a first difference that is the difference between the first relative position information and the second relative position information. The generation unit 213 generates content data that represents a projection image to be projected from the projector 10 into the projection area SR where the projection target SC is installed, based on the first difference. This aspect also makes it possible to generate a projection image that is not intended by the user, based on a change in the relative positional relationship between the projection target SC and the projector 10.

[0030] Furthermore, the first acquisition unit 211 may acquire third relative position information indicating the relative positional relationship between the projection target SC and the projector prior to acquiring the first relative position information, and the second acquisition unit 212 may acquire a second difference, which is the difference between the first relative position information and the third relative position information. The generation unit 213 may then generate content data based on the first difference and the second difference. According to this embodiment, a projection image not intended by the user can be generated and displayed based on time-series difference information such as the first difference and the second difference. Similarly, when an imaging device is used as the measuring instrument 30, a projection image not intended by the user can be generated and displayed based on time-series difference information.

[0031] 3. Variations The above embodiments can be modified as follows. (1) In the above embodiment, the projected image GA is generated by an image generation AI, but any generation condition and an algorithm for generating an image in accordance with the generation condition may be used, and the present invention is not limited to an image generation AI. Furthermore, the generation unit 213 may generate audio data corresponding to the difference acquired by the second acquisition unit 212, in addition to the content data. The display control unit 214 may display the content image and output audio represented by the audio data generated by the generation unit 213.

[0032] (2) To enable image generation that takes into account the user's intentions, the image generator 20 may include a device that allows input of information indicating the user's intentions, and the generation conditions may include conditions related to the user's intentions. The following devices may be incorporated into the configuration. Specific examples of devices that allow input of information indicating the user's intentions include a voice input device (guiding generation by voice), a character input device (guiding generation by language), an image or video input device (guiding generation by visual data), or a pressure sensor (guiding generation by tactile data). By including the user's intentions in the input, it becomes possible to generate a projected image GA that is more in line with the user's intentions.

[0033] (3) In the above embodiment, the first acquisition unit 211, the second acquisition unit 212, the generation unit 213, and the display control unit 214 are software modules. However, any one, any two, any three, or all of the first acquisition unit 211, the second acquisition unit 212, the generation unit 213, and the display control unit 214 may be a hardware module such as an ASIC (Application Specific Integrated Circuit). Even if at least one of the first acquisition unit 211, the second acquisition unit 212, the generation unit 213, and the display control unit 214 is a hardware module, the same effects as those of the above embodiment can be achieved.

[0034] (4) The program PRA may be manufactured as a standalone program or provided free of charge or for a fee. Specific examples of providing the program PRA include providing the program PRA by writing it to a computer-readable recording medium such as a flash ROM, or by downloading it via a telecommunications line such as the Internet. Operating a general computer in accordance with the program PRA provided in these ways enables the computer to execute the display method of the present disclosure. In the above embodiment, the measuring instrument 30 is a device separate from the projector 10 and the image generator 20. However, the measuring instrument 30 may be included in the projector 10 or the image generator 20.

[0035] 4. Summary of this disclosure The present disclosure is not limited to the above-described embodiments and modifications, and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following forms. The technical features in the above embodiments corresponding to the technical features in each form described below can be replaced or combined as appropriate to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. A summary of this disclosure is provided below.

[0036] (Appendix 1) A display method according to the present disclosure includes acquiring first environmental information representing an environment in a projection area onto which an image is projected by a projector, acquiring second environmental information representing the environment at a time later than the time the first environmental information was acquired, acquiring a first difference between the first environmental information and the second environmental information, generating a projection image to be projected from the projector onto the projection area based on the first difference, and displaying the projection image in the projection area by projecting the projection image from the projector onto the projection area. According to the display method of this aspect, the projection image is generated based on the difference between the first environmental information representing the environment in the projection area, where a projection target onto which an image is projected by the projector is installed, and the second environmental information representing the environment at a time later than the time the first environmental information was acquired, so that a projection image not intended by the user can be generated and displayed.

[0037] (Appendix 2) In a more preferred embodiment of the display method, in the display method described in (Supplementary Note 1), the first environmental information is a first captured image obtained by capturing an image of the projection area with an imaging device, and the second environmental information is a second captured image obtained by capturing an image of the projection area with the imaging device at a second time that is later than the first time at which the first captured image is captured. According to the display method of this embodiment, a projection image that is not intended by the user can be generated and displayed based on the difference between the first captured image obtained by capturing an image of the projection area with an imaging device and the second captured image captured at a time later than the first captured image.

[0038] (Appendix 3) In a further preferred embodiment of the display method, in the display method described in (Supplementary Note 2), the first captured image does not include an image projected from the projector, and the second captured image does not include an image projected from the projector. According to this embodiment, even if there is a change in the environment of the projection area while no projection image is being displayed in the projection area, it is possible to generate and display a projection image that is not intended by the user.

[0039] (Appendix 4) Another preferred embodiment of the display method is the display method described in (Supplementary Note 3), and includes: executing a first startup process to start the imaging device; capturing the first captured image after executing the first startup process; capturing the second captured image after capturing the first captured image; executing a second startup process to start the projector after executing the first startup process; and displaying the projection image in the projection area after executing the second startup process. According to this embodiment, by starting the imaging device first when the projector starts up, it is possible to generate and display a projection image that is not intended by the user.

[0040] (Appendix 5) In another preferred embodiment of the display method, in the display method described in any one of (Supplementary Note 1) to (Supplementary Note 4), generating the projection image includes recognizing a change portion from the environment represented by the first environmental information to the environment represented by the second environmental information based on the first difference, and generating an image corresponding to the change portion. According to this embodiment, it is possible to generate a projection image corresponding to the change portion from the environment represented by the first environmental information to the environment represented by the second environmental information.

[0041] (Appendix 6) According to another aspect of the present disclosure, a display method includes: acquiring first relative position information representing a relative positional relationship between a projection target placed in a projection area onto which an image is projected by the projector and the projector; acquiring second relative position information representing a relative positional relationship between the projection target and the projector at a time after the time at which the first relative position information is acquired; acquiring a first difference that is a difference between the first relative position information and the second relative position information; generating a projection image to be projected from the projector onto the projection target based on the first difference; and projecting the projection image from the projector onto the projection area, thereby displaying the projection image in the projection area. According to this aspect, a projection image not intended by a user can be generated and displayed based on the first difference that is a difference between the first relative position information and the second relative position information.

[0042] (Appendix 7) In another preferred embodiment, the display method is the display method described in (Supplementary Note 6), further including: acquiring third relative position information indicating a relative positional relationship between the projection target object and the projector, prior to acquiring the first relative position information; and acquiring a second difference that is a difference between the first relative position information and the third relative position information, wherein generating a projection image to be projected from the projector onto the projection area based on the first difference includes generating the projection image based on the first difference and the second difference. According to this embodiment, a projection image not intended by the user can be generated and displayed based on time-series difference information. [Explanation of symbols]

[0043] 1...system, 10...projector, 20...image generator, 30...measuring instrument, 210...processing device, 211...first acquisition unit, 212...second acquisition unit, 213...generation unit, 214...display control unit, 220...communication device, 230...storage device, PRA...program, MDL...learning model file.

Claims

1. acquiring first environment information representing an environment in a projection area onto which an image is projected by a projector; acquiring second environmental information representing the environment at a time later than a time point at which the first environmental information is acquired; obtaining a first difference between the first environmental information and the second environmental information; generating a projection image to be projected from the projector onto the projection area based on the first difference; projecting the projection image from the projector onto the projection area, thereby displaying the projection image in the projection area; Display methods including.

2. the first environmental information is a first captured image obtained by capturing an image of the projection area with an imaging device, the second environmental information is a second captured image obtained by capturing an image of the projection area by the imaging device at a second time that is later than a first time at which the first captured image is captured; The display method according to claim 1 .

3. the first captured image does not include an image projected from the projector, the second captured image does not include an image projected from the projector; The display method according to claim 2.

4. executing a first startup process for starting up the imaging device; After executing the first startup process, capturing the first captured image; capturing the second captured image after capturing the first captured image; After executing the first startup process, executing a second startup process to start up the projector; After executing the second startup process, displaying the projection image in the projection area; The display method according to claim 3, comprising:

5. generating the projection image includes recognizing a change portion from the environment represented by the first environmental information to the environment represented by the second environmental information based on the first difference, and generating an image according to the change portion; The display method according to claim 1 .

6. the first environment information is first relative position information that indicates a relative positional relationship between a projection target object installed within the projection area and the projector, the second environment information is second relative position information that represents a relative positional relationship between the projection target object and the projector at a time after the time when the first relative position information is acquired; The display method according to claim 1 .

7. acquiring third relative position information that indicates a relative positional relationship between the projection target object and the projector prior to acquiring the first relative position information; acquiring a second difference that is a difference between the first relative position information and the third relative position information, generating a projection image to be projected from the projector onto the projection area based on the first difference includes generating the projection image based on the first difference and the second difference; The display method according to claim 6.

Images