Display method

Abstract

To combine an image of content to be projected onto a projection target on the basis of information which cannot be obtained from a picked-up image of the projection target.SOLUTION: A video generator 20 comprises a first acquisition section 211, a second acquisition section 212, a third acquisition section 213, a fourth acquisition section 214, a generation section 215, and a display control section 216. The first acquisition section 211 acquires a picked-up image by picking up a white image which is projected onto a projection region. The second acquisition section 212 acquires environment information relating to an environment in which a projection target is placed. The third acquisition section 213 acquires physical information on the basis of the picked-up image. The fourth acquisition section 214 acquires user input information representing an instruction of a user. The generation section 215 generates an image of content on the basis of the physical information, the environment information, the user input information, and a learning model file MDL. The display control section 216 causes a projector to project the image of the content.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a display method. [Background technology]

[0002] Patent Document 1 discloses that a projection image is generated by combining an input image with a predetermined background image. Patent Document 1 also discloses that the background image is either prepared in advance or generated by a background image generation unit 33 based on the signal of the input image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-84001 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, a projection image is generated by combining an input image onto a predetermined background image, but it is not possible to combine the image of the content to be projected onto the projection object based on information about the environment in which the projection object is placed, which information cannot be obtained from a captured image of the projection object. [Means for solving the problem]

[0005] One aspect of the display method of the present disclosure includes acquiring a captured image by capturing an image of a projection area, acquiring environmental information that is information about the environment in which a projection object is placed, acquiring physical information based on the captured image, generating a first projection image based on the physical information and the environmental information, and displaying the first projection image. [Brief explanation of the drawings]

[0006] [Figure 1]1 is a diagram illustrating an example configuration of a system 1 according to an embodiment of the present disclosure. [Figure 2] 2 is a diagram showing an example of the configuration of a video generator 20 included in the system 1. FIG. [Figure 3] 2 is a diagram showing an example of an image of content generated by the image generator 20. FIG. [Figure 4] 2 is a diagram showing an example of an image of content generated by the image generator 20. FIG. [Figure 5] 10 is a flowchart showing the flow of processing in a display method executed by a processing device 210 of an image generator 20 in accordance with a program PRA. DETAILED DESCRIPTION OF THE INVENTION

[0007] The embodiments described below are subject to various technically preferable limitations, but the embodiments of the present disclosure are not limited to the following embodiments. 1. Embodiment

[0008] Fig. 1 is a diagram illustrating an example configuration of a system 1 according to an embodiment of the present disclosure. The system 1 is a system that realizes projection mapping by projecting an image onto a projection target. As shown in Fig. 1, the system 1 includes a projector 10, an image generator 20, an image capture device 30, and a user terminal 40. Each of the projector 10, the image capture device 30, and the user terminal 40 is connected to the image generator 20 wirelessly or via a wired connection.

[0009] The projector 10 achieves projection mapping by projecting an image represented by image data supplied from the image generator 20 onto a projection target. That is, in this embodiment, the surface of the projection target is the projection area. The projection target in this embodiment is, for example, a speaker SP installed in the living room of a user's residence, but it may also be a building or the like. The imaging device 30 is, for example, a video camera. The imaging device 30 is placed at a position and orientation such that the projection area of the projection target falls within the imaging field of view. The imaging device 30 captures an image of the projection target under the control of the image generator 20. The imaging device 30 supplies image data representing the captured image obtained by capturing an image of the projection target to the image generator 20.

[0010] The user terminal 40 is, for example, a smartphone. The user terminal 40 accepts operations by a user using the system 1 and transmits user input information corresponding to the operations to the image generator 20. The user input information in this embodiment represents a character string (e.g., vivid or traditional) that is a written instruction from the user regarding the content to be projected onto the projection target. The character string that is a written instruction from the user regarding the content to be projected onto the projection target is an example of first language data in the present disclosure. The user input information reflects the user's intention regarding the content to be projected onto the projection target, and by transmitting this user input information from the user terminal 40 to the image generator 20, the user's intention regarding the content to be projected onto the projection target is transmitted to the image generator 20.

[0011] The image generator 20 is, for example, a personal computer. The image generator 20 generates projection image data representing a projection image to be projected from the projector 10 onto a projection target based on captured image data provided by the imaging device 30 and environmental information provided by the user terminal 40. The environmental information in this embodiment is sound information representing sounds around the projection target. If the user terminal 40 includes a microphone, the user uses the microphone to collect sounds around the projection target and uses the sound information representing the sounds as environmental information. In this embodiment, the microphone of the user terminal 40 is used as a sensor for collecting environmental information. However, the sensor may be provided in the system 1 separately from the user terminal 40. The environmental information is not limited to sound information representing sounds around the projection target. It may also be information representing the amount of external light around the projection target, information representing the temperature, odor, or pressure around the projection target, information representing the amount of vibration in the space in which the projection target is installed, information representing the depth of the space in which the projection target is installed, or information representing the emotions, body temperature, or behavior of people around the projection target. A suitable sensor may be used depending on the type of environmental information.

[0012] FIG. 2 is a diagram showing an example configuration of the image generator 20. As shown in FIG. 2, the image generator 20 includes a processing device 210, a communication device 220, and a storage device 230. The processing device 210 is one or more processors. The processing device 210 is, for example, a CPU (Central Processing Unit). The processing device 210 functions as the control center of the image generator 20 by operating in accordance with a program PRA stored in the storage device 230. The communication device 220 is a device that performs wireless or wired communication with other devices, and includes, for example, an interface circuit. Specific examples of other devices that communicate with the communication device 220 include the projector 10, the image capture device 30, and the user terminal 40.

[0013] The storage device 230 is a recording medium readable by the processing device 210. The storage device 230 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Radom Access Memory). Various programs and a learning model file MDL are stored in the nonvolatile memory of the storage device 230.

[0014] The learning model file MDL is data corresponding to an inference model that has learned the correspondence between image generation conditions (prompts or hyperparameters) and images generated according to the generation conditions. Well-known techniques may be appropriately adopted for the learning model file MDL. In this embodiment, the processing device 210 uses the learning model file MDL to function as an image generation AI that performs comprehensive data analysis and generates images based on information input from multiple modalities (forms) such as images, sounds, and languages.

[0015] Examples of various programs stored in the non-volatile memory include a kernel program and a program PRA. The kernel program is not shown in Figure 2. The kernel program is a program that causes the processing device 210 to implement an OS (Operating System).

[0016] When the image generator 20 is powered on, the processing device 210 reads the kernel program from the nonvolatile memory to the volatile memory and starts executing the read kernel program. The processing device 210, which is operating according to the kernel program, starts executing another program when it is instructed to start executing that other program by an operation represented by operation content data received from the user terminal 40 via the communication device 220. For example, when it is instructed to start executing the program PRA, the processing device 210 reads the program PRA from the nonvolatile memory to the volatile memory and starts executing the program PRA read into the volatile memory.

[0017] Processing device 210 operating in accordance with program PRA functions as first acquisition unit 211, second acquisition unit 212, third acquisition unit 213, fourth acquisition unit 214, generation unit 215, and display control unit 216 shown in Fig. 2. In other words, each of first acquisition unit 211, second acquisition unit 212, third acquisition unit 213, fourth acquisition unit 214, generation unit 215, and display control unit 216 shown in Fig. 2 is a software module realized by operating processing device 210 in accordance with program PRA. The roles of each of first acquisition unit 211, second acquisition unit 212, third acquisition unit 213, fourth acquisition unit 214, generation unit 215, and display control unit 216 shown in Fig. 2 are as follows.

[0018] The first acquisition unit 211 controls the projector 10 to project, for example, a solid white image (second projection image) from the projector 10 onto the projection target, and controls the imaging device 30 to capture the image projected in the projection area and acquire captured image data representing the captured image. Note that when the projection area is designated by the user, the first acquisition unit 211 may also control the imaging device 30 to capture an image of a projection area where the image is not projected and acquire captured image data representing the captured image.

[0019] The second acquisition section 212 communicates with the user terminal 40 to acquire environmental information, which is information relating to the environment in which the projection target is placed and which is not acquired by capturing an image using the image capturing device 30.

[0020] The third acquisition unit 213 acquires physical information based on the captured image represented by the captured image data acquired by the first acquisition unit 211. The physical information is information that represents at least one of the shape, color, and reflection characteristics of the space in which the projection target is installed or the projection target. Specific examples of the information that represents the color and reflection characteristics include the RGB values and brightness of an area in the captured image that corresponds to the projection target.

[0021] The fourth acquisition unit 214 acquires user input information by communicating with the user terminal 40. As described above, the user input information in this embodiment represents a character string that is a written instruction from the user regarding the content to be projected onto the projection target, but it may also represent a voice reading out the instruction, or an image drawn by the user, or a piece of music played or sung by the user. In short, the user input information may be data that reflects the user's intention regarding the content to be projected onto the projection target.

[0022] The generation unit 215 generates content data representing an image of content to be projected from the projector 10 onto the projection target (first projection image) based on the physical information, environmental information, user input information, and the learning model file MDL. More specifically, the generation unit 215 first adjusts the amount of information by filtering or the like so that the physical information, environmental information, and user input information can be treated equally, and generates generation conditions for the content data. These generation conditions include weights that represent the strength of the influence that each of the physical information, environmental information, and user input information has on the content data. The generation unit 215 then inputs the generation conditions into the learning model file MDL, thereby generating content data representing content corresponding to the generation conditions.

[0023] For example, suppose the projection target is a speaker SP as shown in FIGS. 3 and 4, and the environmental information indicates whether or not sound is being output from the speaker SP. In this case, the generation unit 215 recognizes that the projection target is a speaker SP based on the appearance of the projection target represented by the physical information. If the environmental information indicates that sound is being output from the speaker SP, the generation unit 215 generates an image of musical notes placed on the surface of the speaker SP as a content image, as shown in FIG. 3. On the other hand, if the environmental information indicates that sound is not being output from the speaker SP, the generation unit 215 generates an image of the character string "MUTE" placed on the surface of the speaker SP as a content image, as shown in FIG. 4. Note that the user input information is reflected in the color and shape of the musical notes or the character string "MUTE." For example, when the user input information is reflected in the color of the musical notes, etc., the generation unit 215 generates an image in which the musical notes, etc. are drawn in red when "Vivid" is specified, and generates an image in which the musical notes, etc. are drawn in bluish-black when "Traditional" is specified.

[0024] The display control unit 216 transmits the content data generated by the generation unit 215 to the projector 10, and causes the projector 10 to project an image represented by the content data, thereby displaying the image.

[0025] Furthermore, the processing device 210, operating in accordance with the program PRA, executes a display method that prominently exhibits features of the present disclosure. Fig. 5 is a flowchart showing the process flow of this display method. As shown in Fig. 5, this display method includes a first acquisition process SA110, a second acquisition process SA120, a third acquisition process SA130, a fourth acquisition process SA140, a generation process SA150, a display control process SA160, and a determination process SA170.

[0026] In the first acquisition process SA110, the processing device 210 functions as a first acquisition unit 211. In the first acquisition process SA110, the processing device 210 controls the projector 10 to project the second projection image from the projector 10 onto the projection target, and controls the imaging device 30 to capture the image projected onto the projection area and acquire captured image data representing the captured image.

[0027] In a second acquisition process SA120 that follows the first acquisition process SA110, the processing device 210 functions as a second acquisition unit 212. In the second acquisition process SA120, the processing device 210 acquires environmental information. Note that the second acquisition process SA120 may be executed prior to the first acquisition process SA110, or may be executed after the execution of a third acquisition process SA130 or a fourth acquisition process SA140, which will be described later.

[0028] In a third acquisition process SA130 subsequent to the second acquisition process SA120, the processing device 210 functions as a third acquisition unit 213. In the third acquisition process SA130, the processing device 210 acquires physical information based on the captured image represented by the captured image data acquired in the first acquisition process SA110.

[0029] In the fourth acquisition process SA140 that follows the third acquisition process SA130, the processing device 210 functions as a fourth acquisition unit 214. In the fourth acquisition process SA140, the processing device 210 acquires user input information by communicating with the user terminal 40. The fourth acquisition process SA140 may be executed before any one or more of the first acquisition process SA110, second acquisition process SA120, and third acquisition process SA130. In short, it is sufficient that the first acquisition process SA110, second acquisition process SA120, third acquisition process SA130, and fourth acquisition process SA140 have all been executed by the time execution of the generation process SA150, which will be described later, is started.

[0030] In the generation process SA150, the processing device 210 functions as a generation unit 215. In the generation process SA150, the processing device 210 generates content data representing an image of content to be projected from the projector 10 onto the projection target based on physical information, environmental information, user input information, and the learning model file MDL. In the display control process SA160 that follows the generation process SA150, the processing device 210 functions as a display control unit 216. In the display control process SA160, the processing device 210 transmits the content data generated in the generation process SA150 to the projector 10 and displays the image represented by the content data by having the projector 10 project the image. By visually checking the content image projected onto the projection target by the projector 10, the user can check the degree of match between the content image and the shape of the projection target, and if the user is dissatisfied with the degree of match, the user can operate the user terminal 40 to instruct regeneration of the content data. When instructing the regeneration of content data, the user edits the generation conditions, such as by increasing the weight of the physical information included in the generation conditions, in order to improve the degree of match between the content image and the shape of the projection target.

[0031] In determination process SA170 following display control process SA160, the processing device 210 determines whether the user has instructed regeneration of the content data. If the user has instructed regeneration of the content data, the determination result of determination process SA170 is "Yes." If the determination result of determination process SA170 is "Yes," the processing device 210 re-executes the processes from generation process SA150 onwards. Note that in the re-executed generation process SA150, content data is generated using the generation conditions edited by the user. In contrast, if the user has not instructed regeneration of the content data, the determination result of determination process SA170 is "No." If the determination result of determination process SA170 is "No," the processing device 210 terminates this display method. Note that a specific example of when the user does not instruct regeneration of the content data is when there is no dissatisfaction with the degree of match between the content image and the shape of the projection target.

[0032] As described above, according to this embodiment, two processes for realizing projection mapping can be automatically performed: content generation and adjusting the generated content into a projection image that closely matches the shape of the projection target. In this embodiment, content is generated based on physical information and environmental information, making it possible to generate content that reflects sounds and other factors surrounding the projection target. Furthermore, according to this embodiment, in the content generation process, a user can provide certain instructions, and content can be generated that corresponds to those instructions. Note that the user's instructions do not need to take the form of a programming language or an image, but can take a wide variety of forms, such as text input or voice input.

[0033] 2. Other embodiments (1) In the above embodiment, if a user is dissatisfied with the degree of matching of the image to the shape of the projection target, the user increases the weight of the physical information used in generating the image, and repeats changing the weight and outputting the image until the image matches the user's preferences, thereby improving the degree of matching of the image to the shape of the projection target. However, it is also possible to improve the degree of matching of the image to the shape of the projection target by extracting feature points from a captured image of the projection target and using the results in combination with physical information and environmental information. Furthermore, the user may revise the first language data to output an image that better matches the user's preferences.

[0034] (2) When generating images using a generative AI, the image generated may not always be what the user desires, and even if the generation conditions are edited, the image may not always be improved. Therefore, the video generator 20 may store all the generation conditions and generation results, visualize them in a visually understandable manner, such as a flow chart, and display them on the user terminal 40. The user can return to the desired conditions in the flow chart to generate content data.

[0035] 3. Variations The above embodiments can be modified as follows. (1) The generation unit 215 may generate audio data based on physical information and environmental information in addition to content data. The display control unit 216 may display a content image and output audio represented by the audio data generated by the generation unit 215. According to this aspect, it is possible to provide the user with not only an image but also audio as content that matches the space in which the image is projected.

[0036] (2) The user's instructions expressed by the first language data do not need to be complete and may contain deficiencies. If the user's instructions expressed by the first language data are incomplete, the generation unit 215 may generate second language data by referencing the user's past input or a previously prepared dataset, etc. to fill in the deficiencies, and generate content images using this second language data, physical information, and environmental information. A specific example of the user's past input or a previously prepared dataset is information representing the user's instructions or intentions that was input when generating video content in the past.

[0037] (3) In the above embodiment, the first acquisition unit 211, the second acquisition unit 212, the third acquisition unit 213, the fourth acquisition unit 214, the generation unit 215, and the display control unit 216 are software modules. However, any one, any two, any three, any four, any five, or all of the first acquisition unit 211, the second acquisition unit 212, the third acquisition unit 213, the fourth acquisition unit 214, the generation unit 215, and the display control unit 216 may be hardware modules such as ASICs (Application Specific Integrated Circuits). Even if at least one of the first acquisition unit 211, the second acquisition unit 212, the third acquisition unit 213, the fourth acquisition unit 214, the generation unit 215, and the display control unit 216 is a hardware module, the same effects as those of the above embodiment can be achieved. Note that the fourth acquisition unit 214 is not necessarily required and may be omitted. In an aspect in which the fourth acquisition section 214 is omitted, the generation section 215 may generate content data based on the physical information and the environmental information.

[0038] (4) The program PRA may be manufactured as a standalone product or provided free of charge or for a fee. Specific examples of providing the program PRA include providing the program PRA by writing it to a computer-readable recording medium such as a flash ROM, or by downloading it via a telecommunications line such as the Internet. Operating a general computer in accordance with the program PRA provided in these ways enables the computer to execute the display method of the present disclosure. In the above embodiment, the user terminal 40 and the image generator 20 are separate devices, but the user terminal 40 may be included in the image generator 20. The imaging device 30 may be included in any of the projector 10, the image generator 20, or the user terminal 40.

[0039] 4. Summary of this disclosure The present disclosure is not limited to the above-described embodiments and modifications, and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following forms. The technical features in the above embodiments corresponding to the technical features in each form described below can be replaced or combined as appropriate to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. A summary of this disclosure is provided below.

[0040] (Appendix 1) The display method of the present disclosure includes acquiring a captured image by capturing an image of a projection area, acquiring environmental information that is information about the environment in which a projection target is placed, acquiring physical information based on the captured image, generating a first projection image based on the physical information and the environmental information, and displaying the first projection image. According to the display method of this aspect, the first projection image is generated based on the physical information and the environmental information, allowing a user to easily project an image related to content.

[0041] (Appendix 2) In a more preferred embodiment of the display method, in the display method described in (Supplementary Note 1), generating the first projection image based on the physical information and the environmental information may include generating the first projection image based on the physical information, the environmental information, and a learning model file. According to the display method of this embodiment, the first projection image is generated based on the physical information, the environmental information, and a learning model file, so that the user can easily project an image related to the content.

[0042] (Appendix 3) A more preferred embodiment of the display method may further include generating audio data based on the physical information and the environmental information, and outputting audio represented by the audio data while displaying the first projection image, in the display method described in (Supplementary Note 1) or (Supplementary Note 2). According to this embodiment, it is possible to provide the user with not only an image but also audio as content that matches the space in which the image is projected.

[0043] (Appendix 4) In another preferred embodiment, the display method of the display method according to (Supplementary Note 1), (Supplementary Note 2), or (Supplementary Note 3) further includes acquiring first language data, and generating the first projection image based on the physical information and the environmental information may include generating the first projection image based on the physical information, the environmental information, and the first language data. According to the display method of this embodiment, a smart speaker or the like can reflect instructions or requests received from a user in content.

[0044] (Appendix 5) In another preferred embodiment, the display method of the display method described in (Supplementary Note 4) further includes complementing the first language data based on a data set to generate second language data, and generating the first projection image based on the first language data may include generating the first projection image based on the second language data. According to this embodiment, if an instruction received from a user is ambiguous, the instruction can be clarified. [Explanation of symbols]

[0045] 1...system, 10...projector, 20...image generator, 30...imaging device, 40...user terminal, 210...processing device, 211...first acquisition unit, 212...second acquisition unit, 213...third acquisition unit, 214...fourth acquisition unit, 215...generation unit, 216...display control unit, 220...communication device, 230...storage device, PRA...program, MDL...learning model file.

Claims

1. acquiring a captured image by capturing an image of the projection area; Acquiring environmental information, which is information about the environment in which the projection target is placed; acquiring physical information based on the captured image; generating a first projection image based on the physical information and the environmental information; displaying the first projected image; Display methods including.

2. generating the first projection image based on the physical information and the environmental information includes generating the first projection image based on the physical information, the environmental information, and a learning model file; The display method according to claim 1 .

3. generating audio data based on the physical information and the environmental information; displaying the first projection image and outputting sound represented by the sound data; The display method of claim 1 further comprising:

4. further comprising obtaining first language data; generating the first projection image based on the physical information and the environmental information includes generating the first projection image based on the physical information, the environmental information, and the first language data; The display method according to any one of claims 1 to 3.

5. further comprising: complementing the first language data based on a dataset to generate second language data; generating the first projection image based on the first language data includes generating the first projection image based on the second language data; The display method according to claim 4.

Images