Display method

Abstract

To generate and display a content corresponding to a geometry included in a real space, such as a point, a line, or a plane.SOLUTION: A video generator 20 acquires a captured image by causing an imaging device 30 to capture an image of a projection object in a real space. The video generator 20 acquires geometry information indicating a geometry selected by a user from among geometries included in a range of the captured image, and acquires instruction information representing an instruction of the user regarding an expression of a content to be displayed in a projection area. The video generator 20 generates, on the basis of the geometry information selected by the user and the instruction information, a first projection image regarding the content for which the expression indicated by the instruction information is implemented with the geometry indicated by the geometry information as a starting point, and causes a projector 10 to project the first projection image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a display method. [Background technology]

[0002] Patent Document 1 discloses that a projection image to be projected onto a selected object is generated by combining an input image with a predetermined background image. Patent Document 1 also discloses that the background image is an image prepared in advance or an image generated by a background image generation unit based on a signal from the input image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-84001 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 does not disclose how to select an object to be projected in a projection image. For example, with the technology disclosed in Patent Document 1, there is a risk that two-dimensional objects such as markers or tape attached to a flat surface may not be selected as a projection target. [Means for solving the problem]

[0005] One aspect of the display method of the present disclosure includes acquiring a captured image by capturing an image of a projection target in real space, acquiring geometry information included within a range of the captured image selected by a user, acquiring information regarding instructions from the user, generating a first projection image representing content to be displayed on the projection target based on the geometry information and the instruction information, and displaying the content by projecting the first projection image into the projection area. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a diagram illustrating an example configuration of a system 1 according to an embodiment of the present disclosure. [Figure 2] 10 is a diagram showing an example of a projection image GA projected onto a projection area SR from a projector 10. FIG. [Figure 3] 2 is a diagram showing an example of the configuration of a video generator 20 included in the system 1. FIG. [Figure 4] FIG. 10 is a diagram showing an example of a selection auxiliary image GB. [Figure 5] 10 is a flowchart showing the flow of processing in a display method executed by a processing device 210 of an image generator 20 in accordance with a program PRA. DETAILED DESCRIPTION OF THE INVENTION

[0007] The embodiments described below are subject to various technically preferable limitations, but the embodiments of the present disclosure are not limited to the following embodiments.

[0008] 1. Embodiment Fig. 1 is a diagram illustrating an example configuration of a system 1 according to an embodiment of the present disclosure. The system 1 is a system that realizes projection mapping by projecting a projection image GA onto a projection target SC. As shown in Fig. 1, the system 1 includes a projector 10, an image generator 20, and an image capture device 30. The projector 10 and the image capture device 30 are each connected to the image generator 20 wirelessly or via a wired connection.

[0009] The projector 10 achieves projection mapping by projecting a projection image GA represented by image data supplied from the image generator 20 onto a projection area SR where a projection target SC is located. In this embodiment, the projection target SC is, for example, a poster, and the projection area SR is one of the walls of a room of a user using the system 1, with the projection target SC attached to that wall. In this embodiment, the projection mapping described above projects a projection image GA that adds a virtual frame VW around the projection target SC, as shown in FIG. 2. Note that in FIG. 2, the projection target SC is depicted with a dotted line. A room with a wall to which the projection target SC is attached is an example of real space in this disclosure, and the projection image GA representing the virtual frame VW is an example of a first projection image in this disclosure. The virtual frame VW is an example of content in this disclosure. Furthermore, although the projection target SC in this embodiment is a poster, it may also be a painting, illustration, or photograph, or a marker or tape attached to the projection area SR. It should be noted that the projection target SC in the present disclosure is not limited to a two-dimensional object such as a poster, but may also be a stereoscopic object having a three-dimensional shape.

[0010] The imaging device 30 is, for example, a camera, which captures an image of the projection area SR and supplies captured image data representing the captured image to the image generator 20. The image generator 20 is, for example, a personal computer. Based on the captured image data supplied from the imaging device 30 and instructions given by the user, the image generator 20 generates content data representing a projection image GA to be projected from the projector 10 onto the projection area SR.

[0011] The image generator 20 communicates with the image recognition server 40 via a network NW such as the Internet. The image recognition server 40 recognizes the type of object (object) present within the range of an image represented by image data received via the network NW, recognizes the surfaces, lines, and points (hereinafter referred to as "geometry") that define the contours of the object, and returns geometry information indicating the recognized geometry for each recognized object to the sender of the image data. Existing technologies may be appropriately adopted as the geometry recognition method used by the image recognition server 40. In this embodiment, the image generator 20 generates content data representing content that uses the selected geometry as a starting point by having the user select one of multiple geometries recognized by the image recognition server 40 and specify how the content will be expressed based on the selected geometry.

[0012] FIG. 3 is a diagram showing an example of the configuration of the image generator 20. As shown in FIG. 3, the image generator 20 includes a processing device 210, a communication device 220, a microphone 230, and a storage device 240. The processing device 210 is one or more processors. The processing device 210 is, for example, a CPU (Central Processing Unit). The processing device 210 functions as the control center of the image generator 20 by operating in accordance with a program PRA stored in the storage device 240. The communication device 220 is a device that performs wireless or wired communication with other devices, and includes, for example, an interface circuit. Specific examples of other devices that communicate with the communication device 220 include the projector 10, the image capture device 30, and the image recognition server 40.

[0013] The microphone 230 picks up the user's voice. The microphone 230 outputs audio information representing the picked up sound to the processing device 210. Although details will be described later, in this embodiment, the selection of geometry that associates content to be projected from the projector 10 onto the projection area SR and the designation of the representation of the content are performed by the user's voice.

[0014] The storage device 240 is a recording medium readable by the processing device 210. The storage device 240 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Radom Access Memory). The nonvolatile memory of the storage device 240 stores various programs and a learning model file MDL.

[0015] The learning model file MDL is data corresponding to an inference model that has learned the correspondence between image generation conditions and images generated in accordance with the generation conditions. Known technologies may be appropriately employed for the learning model file MDL. In this embodiment, the processing device 210 functions as an image generation AI that uses the learning model file MDL to generate an image of content that expresses a user-specified geometry from among multiple geometries included in the projection area SR. The generation conditions in this embodiment are a geometry to which content is associated and a designation of the content's expression. In this embodiment, the learning model file MDL pre-associates, for example, a virtual frame VW with the outline of a poster, and also associates a periodic color change (or periodic width variation) as the expression of the virtual frame VW. As will be described in detail later, the user can specify the geometry to which content is associated by uttering "outline of poster," and can specify the content expression to be associated with the geometry by uttering "periodic color change."

[0016] Examples of various programs stored in nonvolatile memory include a kernel program and a program PRA. The kernel program is not shown in FIG. 3. The kernel program is a program that causes the processing device 210 to implement an operating system (OS). When the image generator 20 is powered on, the processing device 210 reads the kernel program from nonvolatile memory to volatile memory and begins executing the read kernel program. The processing device 210, operating according to the kernel program, begins executing another program when instructed to do so via an input device (not shown). For example, when instructed to start executing the program PRA, the processing device 210 reads the program PRA from nonvolatile memory to volatile memory and begins executing the program PRA read into volatile memory.

[0017] Processing device 210 operating in accordance with program PRA functions as first acquisition unit 211, second acquisition unit 212, third acquisition unit 213, generation unit 214, and display control unit 215 shown in Fig. 3. In other words, each of first acquisition unit 211, second acquisition unit 212, third acquisition unit 213, generation unit 214, and display control unit 215 shown in Fig. 3 is a software module realized by operating processing device 210 in accordance with program PRA. The roles of each of first acquisition unit 211, second acquisition unit 212, third acquisition unit 213, generation unit 214, and display control unit 215 shown in Fig. 3 are as follows.

[0018] The first acquisition unit 211 controls the projector 10 to project a predetermined image, such as a white image, onto the projection area SR, and controls the imaging device 30 to cause the imaging device 30 to capture the projection area SR onto which the predetermined image is projected. The first acquisition unit 211 then acquires captured image data representing the captured image obtained by capturing the predetermined image projected onto the projection area SR. The predetermined image is an image for assisting object recognition by the image recognition server 40, and is an example of a second projected image in the present disclosure.

[0019] The second acquisition unit 212 transmits the captured image data acquired from the imaging device 30 to the image recognition server 40 using the communication device 220, thereby recognizing the types of various objects appearing in the captured image represented by the captured image data and recognizing the surfaces, lines, and points (hereinafter, referred to as geometry) that define the contours of the objects, and acquires geometry information indicating the recognized geometry for each recognized object. This geometry information includes information indicating the position of the geometry indicated by the geometry information in the captured image. Then, based on the captured image represented by the captured image data acquired by the first acquisition unit 211 and the geometry information acquired from the image recognition server 40, the second acquisition unit 212 generates image data representing a selection auxiliary image that emphasizes the geometry indicated by the geometry information in the captured image represented by the captured image data, and projects the image represented by the image data using the projector 10 to highlight the geometry. In this embodiment, the projection target SC appears in the captured image represented by the captured image data acquired by the first acquisition unit 211, and geometry information indicating the contour line of the projection target SC is acquired. As shown in Fig. 4, the contour line is highlighted with, for example, a thick red line BL, and a selection supplementary image GB, in which the area other than the thick line BL is colored white, is projected from the projector 10 toward the projection area SR while being positioned so as to overlap with the projection target SC. Note that in Fig. 4, the projection target SC is drawn with a dotted line, as in Fig. 2. The selection supplementary image GB, like the predetermined image described above, is an example of a second projection image in the present disclosure.

[0020] A user who views the selection assist image GB can grasp at a glance one or more geometries included in the projection area SR. The user selects, by voice, a geometry to which content is to be associated from one or more geometries included in the projection area SR. As described above, in this embodiment, the user specifies the geometry to which content is to be associated by speaking "outline of poster." When this voice is acquired by the microphone 230, the second acquisition unit 212 performs voice recognition on the voice to acquire geometry information indicating the "outline of poster" as geometry information indicating the geometry selected by the user (i.e., the geometry to which content is to be associated).

[0021] The third acquisition unit 213 acquires instruction information including a user's instruction regarding the representation of content to be associated with the geometry selected by the user. As described above, in this embodiment, the user specifies the representation of content to be associated with the selected geometry by speaking "periodic color change." When this voice is acquired by the microphone 230, the third acquisition unit 213 performs voice recognition on the voice to identify "periodic color change" as the representation specified by the user.

[0022] The generation unit 214 uses the image generation AI described above to generate content data representing a projection image GA to be projected from the projector 10 onto the projection area SR, based on the geometry information acquired by the second acquisition unit 212 and the instruction information acquired by the third acquisition unit 213. For example, if the "outline of a poster" is selected as the geometry to which the content is associated, and "periodic color change" is specified as the representation of the content, the generation unit 214 uses the image generation AI described above to generate content data representing a projection image GA that represents a "virtual frame whose color changes periodically."

[0023] The display control unit 215 transmits the content data generated by the generation unit 214 to the projector 10, and causes the projector 10 to project a projection image GA represented by the content data, thereby displaying the projection image GA in the projection area SR. As a result, the projection image GA of the content that executes the expression specified by the user is displayed in the projection area SR, starting from the geometry selected by the user. In this embodiment, a projection image GA representing a virtual frame VW whose color changes periodically is displayed around the projection target SC.

[0024] Furthermore, the processing device 210, operating in accordance with the program PRA, executes a display method that prominently exhibits features of the present disclosure. Fig. 5 is a flowchart showing the process flow of this display method. As shown in Fig. 5, this display method includes a first acquisition process SA110, a second acquisition process SA120, a third acquisition process SA130, a generation process SA140, a display control process SA150, and a determination process SA160.

[0025] In the first acquisition process SA110, the processing device 210 functions as a first acquisition unit 211. In the first acquisition process SA110, the processing device 210 controls the projector 10 to project a predetermined image, such as a white image, onto the projection area SR, and controls the imaging device 30 to image the projection area SR onto which the predetermined image is projected. Then, the processing device 210 acquires captured image data representing the captured image obtained by capturing the predetermined image projected onto the projection area SR.

[0026] In the second acquisition process SA120 subsequent to the first acquisition process SA110, the processing device 210 functions as a second acquisition unit 212. In the second acquisition process SA120, the processing device 210 inputs the captured image data acquired from the imaging device 30 to the image recognition server 40, thereby recognizing the types of various objects appearing in the captured image represented by the captured image data, and also acquires geometry information representing each of the points, lines, and surfaces (hereinafter collectively referred to as geometry) included within the range of the captured image from the image recognition server 40. Next, based on the captured image represented by the captured image data acquired in the first acquisition process SA110 and the geometry information acquired from the image recognition server 40, the processing device 210 generates image data representing a selection auxiliary image GB that emphasizes the geometry indicated by the geometry information in the captured image represented by the captured image data, and causes the projector 10 to project the image represented by the image data, thereby emphasizing the geometry.

[0027] The user who views the selection assist image GB selects, by voice, one or more geometries included in the projection area SR to which the content is to be associated. In the second acquisition process SA120, the processing device 210 acquires audio information representing the user's voice from the microphone 230, thereby identifying the geometry selected by the user.

[0028] In the third acquisition process SA130, the processing device 210 functions as a third acquisition unit 213. In the third acquisition process SA130, the processing device 210 acquires instruction information including a user instruction regarding the representation of the content to be associated with the geometry indicated by the geometry information acquired in the second acquisition process SA120.

[0029] In the generation process SA140, the processing device 210 functions as a generation unit 214. In the generation process SA140, the processing device 210 generates content data representing a projection image GA to be projected from the projector 10 onto the projection area SR using the image generation AI described above, based on the geometry information acquired in the second acquisition process SA120 and the instruction information acquired in the third acquisition process SA130. The content data generated in the generation process SA140 of this embodiment starts from the geometry selected by the user and represents a projection image GA of content that executes an expression specified by the user.

[0030] In display control processing SA150 subsequent to generation processing SA140, the processing device 210 functions as a display control unit 215. In display control processing SA150, the processing device 210 transmits the content data generated in generation processing SA140 to the projector 10, and displays a projection image GA by having the projector 10 project an image represented by the content data. As a result, the projection image GA of the content that executes the expression specified by the user is displayed in the projection area SR, starting from the geometry selected by the user.

[0031] In determination processing SA160 following display control processing SA150, the processing device 210 determines whether or not the user has instructed the device to end the display of the projected image GA. If the user has instructed the device to end the display of the projected image GA, the determination result of determination processing SA160 is "Yes." If the determination result of determination processing SA160 is "Yes," the processing device 210 ends the execution of this display method. In contrast, if the user has not instructed the device to end the display of the projected image GA, the determination result of determination processing SA160 is "No." If the determination result of determination processing SA160 is "No," the processing device 210 re-executes the first acquisition processing SA110 and subsequent processing.

[0032] As described above, according to this embodiment, even if the projection target SC is a two-dimensional object, the user can select the contour line of the object as the geometry to which the content is to be associated and specify the representation of the content, thereby causing a projection image GA of the content that starts from the geometry and executes the operation to be displayed in the projection area SR.

[0033] 2. Other embodiments (1) In the above embodiment, instructions regarding the content expression are input by the user's voice, but they may also be input by the user's gesture. For example, a table associating user gestures with content expressions may be stored in advance in the image generator 20, and the processing device 210 may identify the user's gesture by performing image recognition on the captured image obtained by capturing an image of the user specifying the content expression with a gesture using the imaging device 30, and then identify the content expression based on the identified gesture and the table.

[0034] (2) In the above embodiment, the user selected a geometry to which content is to be associated. However, the user may also select a geometry to which content is not to be associated. In this case, content data is generated by the generation unit 214 for the projection area SR excluding the geometry selected by the user. According to this aspect, by specifying a geometry to which content is not to be associated, it becomes possible to generate and display content corresponding to real space. Also, in the above embodiment, the geometry included in the projection area SR is highlighted by displaying the selection assist image GB. However, this highlighting is not essential and may be omitted.

[0035] (3) The geometry to which the content is to be associated may be selected by the user touching the corresponding geometry in the projection area SR with a pointer, such as a finger or a pointer wand. In this case, the geometry to which the content is to be associated may be identified by analyzing a captured image of the projection area SR in a state in which the geometry to which the content is to be associated has been specified by the user. The pointer may be a virtual pointer, such as a laser beam emitted from a laser pointer. Furthermore, if the image generator 20 includes a touch panel consisting of a display device, such as a liquid crystal display, and a touch sensor covering the display surface of the display device, the captured image of the projection area SR may be displayed on the display device, and the user may select the geometry to which the content is to be associated by performing a touch operation, such as tracing a finger or the like, over any of the geometries displayed on the display device. This embodiment also allows the user to directly select the geometry to which the content is to be associated.

[0036] 3. Variations The above embodiments can be modified as follows. (1) The generation unit 214 may generate, in addition to the content data, audio data according to the geometry indicated by the geometry information acquired by the second acquisition unit 212 and the instruction indicated by the instruction information acquired by the third acquisition unit 213. The display control unit 215 may display the content image and output audio represented by the audio data generated by the generation unit 214.

[0037] (2) In the above embodiment, the first acquisition unit 211, the second acquisition unit 212, the third acquisition unit 213, the generation unit 214, and the display control unit 215 are software modules. However, any one, any two, any three, any four, or all of the first acquisition unit 211, the second acquisition unit 212, the third acquisition unit 213, the generation unit 214, and the display control unit 215 may be a hardware module such as an ASIC (Application Specific Integrated Circuit). Even if at least one of the first acquisition unit 211, the second acquisition unit 212, the third acquisition unit 213, the generation unit 214, and the display control unit 215 is a hardware module, the same effects as those of the above embodiment can be achieved.

[0038] (4) The program PRA may be manufactured as a standalone product or provided free of charge or for a fee. Specific examples of providing the program PRA include providing the program PRA by writing it to a computer-readable recording medium such as a flash ROM, or by downloading it via a telecommunications line such as the Internet. Operating a general computer in accordance with the program PRA provided in these ways enables the computer to execute the display method of the present disclosure. In the above embodiment, the image capture device 30 is a device separate from the projector 10 and the image generator 20. However, the image capture device 30 may be included in the projector 10 or the image generator 20.

[0039] 4. Summary of this disclosure The present disclosure is not limited to the above-described embodiments and modifications, and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following forms. The technical features in the above embodiments corresponding to the technical features in each form described below can be replaced or combined as appropriate to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. A summary of this disclosure is provided below.

[0040] (Appendix 1) A display method according to the present disclosure includes: acquiring a captured image by capturing an image of a projection target in real space; acquiring geometry information indicating a point, line, or surface selected by a user from among points, lines, or surfaces included within the captured image; acquiring instruction information indicating an instruction from the user regarding content to be displayed in the projection area; generating a first projection image of the content based on the geometry information and the instruction information; and projecting the first projection image onto the projection area where the projection target is installed, thereby displaying the content. According to the display method of this aspect, content corresponding to real space can be generated and displayed.

[0041] (Appendix 2) A more preferred aspect of the display method is the display method described in (Supplementary Note 1), in which the instruction information includes an instruction regarding the expression of the content, and the first projected image represents the content that executes the expression indicated by the instruction information, starting from a point, line, or plane represented by the geometry information. According to this aspect, it is possible to generate and display content that corresponds to real space and executes the expression instructed by the user, starting from geometry in the real space.

[0042] (Appendix 3) A further preferred embodiment of the display method is the display method described in (Supplementary Note 2), in which the user instructs the representation of the content by a gesture, and the instruction information is obtained by capturing an image of the user's gesture. According to this embodiment, the user can instruct, by a gesture, the representation of the content that corresponds to real space and executes the representation using geometry in the real space as a starting point.

[0043] (Appendix 4) In another preferred embodiment of the display method, in the display method described in (Supplementary Note 1), generating the first projection image includes generating the first projection image for a projection area excluding points, lines, or surfaces indicated by the geometry information. According to this embodiment, a user can specify an area to which the user does not want content to be associated by selecting geometry.

[0044] (Appendix 5) Another preferred embodiment of the display method is the display method according to any one of (Supplementary Note 1) to (Supplementary Note 3), in which the selection of points, lines, or surfaces included within the captured image is performed based on the user's voice. According to this embodiment, the user can select geometry information by voice, eliminating the need for input operations using fingers.

[0045] (Appendix 6) In addition, in another aspect of the present disclosure, there is provided a display method according to any one of (Supplementary Note 1) to (Supplementary Note 3), wherein the selection of a point, line, or surface included within the range of the captured image is performed by pointing a point, line, or surface within the projection area with a pointer in the real space. According to this aspect, the user can directly select geometry.

[0046] (Appendix 7) Another preferred embodiment of the display method is the display method according to any one of (Supplementary Note 1) to (Supplementary Note 3), in which the selection of a point, line, or surface included within the captured image is performed by a touch operation on a display surface of a display device that displays the captured image. This embodiment also allows the user to directly select geometry.

[0047] (Appendix 8) Another preferred embodiment of the display method is the display method according to any one of (Supplementary Note 1) to (Supplementary Note 6), further comprising: identifying points, lines, or surfaces included in the projection area based on the captured image; and projecting a second projection image onto the projection area, the second projection image highlighting the identified points, lines, or surfaces, prior to acquiring the geometry information. According to this embodiment, the geometry recognized based on the captured image can be presented to the user in an easy-to-understand manner. [Explanation of symbols]

[0048] 1...system, 10...projector, 20...image generator, 30...imaging device, 40...image recognition server, 210...processing device, 211...first acquisition unit, 212...second acquisition unit, 213...third acquisition unit, 214...generation unit, 215...display control unit, 220...communication device, 230...microphone, 240...storage device, PRA...program, MDL...learning model file.

Claims

1. acquiring a captured image by capturing an image of a projection target in real space; Obtaining geometry information indicating a point, line, or surface selected by a user from among points, lines, or surfaces included within the captured image; acquiring instruction information representing an instruction from the user regarding content to be displayed in a projection area where the projection target is installed; generating a first projection image relating to the content based on the geometry information and the instruction information; projecting the first projection image onto the projection area to display the content; Display methods including.

2. the instruction information includes instructions regarding the presentation of the content; The first projected image is The content executes the expression indicated by the instruction information, starting from a point, a line, or a surface represented by the geometry information. The display method according to claim 1 .

3. the user indicates the presentation of the content through a gesture; The instruction information is obtained by capturing an image of the user's gesture. The display method according to claim 2.

4. generating the first projected image includes: generating the first projection image for a projection area excluding the points, lines, or surfaces indicated by the geometry information; The display method according to claim 1 .

5. The selection of points, lines, or surfaces included within the captured image is performed based on the user's voice. The display method according to claim 1 .

6. The selection of a point, a line, or a surface included within the range of the captured image is performed by pointing a point, a line, or a surface within the projection area with a pointer in the real space. The display method according to claim 1 .

7. The selection of a point, a line, or a surface included within the range of the captured image is performed by a touch operation on a display surface of a display device that displays the captured image. The display method according to claim 1 .

8. Identifying points, lines, or surfaces included in the projection area based on the captured image; and projecting a second projection image onto the projection area, the second projection image highlighting the identified points, lines, or surfaces, prior to obtaining the geometry information. The display method according to claim 1 .

Images