Program, content editing method, and information processing device

Abstract

To shorten processing time required for content editing.SOLUTION: A program causes a computer to: execute first processing of obtaining measurement information used for adjusting a projection image that is projected in a projection area, based on a plurality of captured images obtained by imaging each of a plurality of pattern images that are sequentially projected in the projection area from a projector; and execute, during execution of the first processing, second processing that is processing different from the first processing and relates to the projection image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a program, a content editing method, and an information processing device. [Background technology]

[0002] In projection mapping, which uses equipment such as a projector to project an image onto a three-dimensional object, the position, shape, and size of the image are generally adjusted to match the position, shape, and size of the object.

[0003] For example, Patent Document 1 describes a method of generating a captured image by capturing at least a part of a projection image projected by a projector onto a projection target, and setting at least one of the position and size of the projection image in the captured image. This setting uses measurement results based on a plurality of captured images capturing each of a plurality of pattern images, such as binary codes, sequentially projected onto a projection surface from a projector. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-158625 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, it is necessary to repeat the projection and capture of the pattern image a number of times according to the display resolution of the projector. In the technology described in Patent Document 1, other processing is executed after the measurement. Since the execution of other processing cannot be accepted during measurement, there is an issue that it takes a long time from the start of measurement to the completion of the setting of the projection image. [Means for solving the problem]

[0006] A program according to one embodiment of the present disclosure causes a computer to execute a first process to obtain measurement information used to adjust a projection image projected onto a projection area based on a plurality of captured images each of a plurality of pattern images that are sequentially projected onto a projection area from a projector, and execute a second process related to the projection image, which is different from the first process, during the period in which the first process is being executed.

[0007] A content editing method according to one embodiment of the present disclosure includes executing a first process to obtain measurement information used to adjust a projection image projected onto a projection area based on a plurality of captured images each of which is a plurality of pattern images that are sequentially projected onto a projection area from a projector, and executing, during a period in which the first process is being executed, a second process related to the projection image, which is different from the first process.

[0008] An information processing device according to one embodiment of the present disclosure includes a processing device that performs a first process to obtain measurement information used to adjust a projection image projected onto a projection area based on a plurality of captured images each of which is an image of a plurality of pattern images that are sequentially projected onto a projection area from a projector, and performs a second process related to the projection image, which is different from the first process, during a period in which the first process is being performed. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing an overview of a system used in a content editing method according to a first embodiment. [Diagram 2] 1 is a block diagram of an information processing device according to a first embodiment. [Diagram 3] 4 is a flowchart showing a flow of a content editing method according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining installation of a projector. [Diagram 5] FIG. 11 is a diagram for explaining an instruction to start execution of a first process. [Figure 6]FIG. 11 is a diagram for explaining selection of a content image. [Figure 7] FIG. 13 is a diagram showing a display example of a content image superimposed on an initial captured image. [Figure 8] FIG. 13 is a diagram showing a display example after a first process. [Figure 9] FIG. 13 is a diagram showing a display example after a first process. [Figure 10] FIG. 11 is a diagram for explaining editing of a content image after the first processing. [Figure 11] FIG. 4 is a diagram for explaining generation of a projection image. [Figure 12] FIG. 13 is a diagram showing an example of a display during generation of a projection image. [Figure 13] FIG. 13 is a diagram showing an example of a display when generation of a projection image is completed. [Figure 14] 13 is a diagram for explaining projection using an edited content image. FIG. [Figure 15] 13 is a flowchart showing the flow of a content editing method according to the second embodiment. [Figure 16] FIG. 13 is a diagram showing an example of a display of a content image superimposed on a line drawing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions, and some parts are shown diagrammatically to facilitate understanding. In addition, the scope of the present disclosure is not limited to these forms unless otherwise specified in the following description to the effect that the present disclosure is limited. 1. First embodiment 1-1. Overview of the system used for the content editing method 1 is a diagram showing an outline of a system 100 used in a content editing method according to the first embodiment. The system 100 is a projection mapping system that projects an image so as to match the shape of an object OJa, which is a projection target.

[0011] In the example shown in FIG. 1, the object OJa is a plain T-shirt, such as white, worn by the object OJb. The object OJb is, for example, a torso or a mannequin. The shapes, sizes, positions, and other aspects of the objects OJa and OJb are not limited to the example shown in FIG. 1 and are arbitrary. The object OJa is not limited to a T-shirt and is arbitrary as long as it is an object onto which projection mapping is performed. The object OJb is not limited to a torso or a mannequin and may be used or omitted as necessary.

[0012] As shown in Fig. 1, the system 100 includes a camera 10, a projector 20, and an information processing device 30. Each unit of the system 100 will be briefly described below with reference to Fig. 1.

[0013] The camera 10 is a digital camera having an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 10 generates captured image data DGa, which will be described later, indicating captured images of objects OJa and OJb. Here, the objects OJa and OJb are included in an imaging region RC, which is an area that the camera 10 can capture.

[0014] In the example shown in FIG. 1, the imaging region RC includes an object OJc in addition to the objects OJa and OJb. The object OJc is a screen installed along a wall W located behind the objects OJa and OJb when viewed from the camera 10 and the projector 20. The object OJc may be used as necessary and may be omitted. The installation position and installation posture of the camera 10 are not limited to the example shown in FIG. 1 and may be arbitrary. Furthermore, the camera 10 may be a part of the information processing device 30.

[0015] The projector 20 is a display device that projects an image onto an object OJa under the control of an information processing device 30. Here, a projection area RP, which is an area where the projector 20 can project, includes the objects OJa, OJb, and OJc.

[0016] In the example shown in Fig. 1, the projection area RP is included in the imaging area RC. The projection area RP only needs to include the objects OJa and OJb, and may have a portion that is not included in the imaging area RC. The installation position and installation posture of the projector 20 are not limited to the example shown in Fig. 1, and are arbitrary.

[0017] Although not shown, the projector 20 has an image processing circuit, a light source, a light modulation device, and a projection optical system. The image processing circuit of the projector 20 is a circuit that controls the driving of the light modulation device of the projector 20 based on information from the information processing device 30. The light source of the projector 20 includes, for example, a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, and emits red, green, and blue light, respectively. The light modulation device of the projector 20 includes three light modulation elements provided corresponding to red, green, and blue. Each of the three light modulation elements is, for example, a display panel such as a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (digital mirror device). The three light modulation elements modulate the red, green, and blue light, respectively, based on a signal from the image processing circuit of the projector 20 to generate image light of each color. Such image light of each color is synthesized by a color synthesis optical system to become full-color image light. The projection optical system of the projector 20 is an optical system including a projection lens and the like, and forms and projects the above-mentioned full-color image light onto a projection target.

[0018] The information processing device 30 is a computer that executes a content editing method that will be described in detail later. The information processing device 30 has a function of controlling the operations of the camera 10 and the projector 20, a function of executing a first process S1 that measures the projection surface of the projector 20 using the camera 10 and the projector 20, and a function of editing an image to be used for projection by the projector 20 using the result of the measurement and a captured image acquired from the camera 10.

[0019] 1, the information processing device 30 is a notebook computer. The information processing device 30 is not limited to a notebook computer, and may be, for example, a desktop computer, a smartphone, a tablet terminal, or the like. In addition, when the information processing device 30 has an imaging function, the information processing device 30 may also function as the camera 10.

[0020] As will be described later in detail, the information processing device 30 has a display device 34 and an input device 35, and displays a user interface image GU, which is a GUI (graphical user interface) image required to execute the content editing method, on the display device 34. In the drawings, the "user interface" may be written as "UI". Then, the information processing device 30 executes editing of an image to be used for projection by the projector 20 based on an instruction from the user via the input device 35. The user interface image GU can receive an editing operation from the user to match the shape of the image projected from the projector 20 to the projection target with the shape of the projection target. Here, the user interface image GU can receive an instruction to execute a first process for measuring the projection surface of the projector 20, and can receive an instruction for a second process S2 different from the first process S1 during the execution of the first process.

[0021] 1-2. Information processing device Fig. 2 is a block diagram of an information processing device 30 according to the first embodiment. As shown in Fig. 2, the information processing device 30 includes a storage device 31, a processing device 32, a communication device 33, a display device 34, and an input device 35. These are connected to each other so as to be able to communicate with each other.

[0022] The storage device 31 is a storage device that stores programs such as an operating system and application programs executed by the processing device 32, and data processed by the processing device 32. The storage device 31 includes, for example, a hard disk drive or a semiconductor memory. Note that a part or all of the storage device 31 may be an external storage device of the information processing device 30, or may be provided in an external device such as a server connected to the information processing device 30 via a communication network such as the Internet.

[0023] The program PR, captured image data DGa, line drawing data DGb, content image data DGc, edit image data DGd, projection image data DGe, and conversion data DGf are stored in the storage device 31. The conversion data DGf is an example of "measurement information".

[0024] The program PR is a program for executing a content editing method described later in detail. The captured image data DGa is data indicating a captured image acquired from the camera 10. The captured image data DGa is data indicating a plurality of captured images obtained during measurement by the measurement unit 32e described later, and includes initial image data DGa0 indicating a captured image obtained by capturing an initial state projection area RP. The initial image data DGa0 is obtained by capturing an image of the projection area RP on which nothing is projected, or by capturing an image of the projection area RP when a black image or a white image included in a plurality of pattern images described later is projected onto the projection area RP. However, the captured image data DGa may include data indicating a captured image separate from the captured image obtained during measurement by the measurement unit 32e described later. The line drawing data DGb is data in which the captured image indicated by the captured image data DGa is expressed by a line drawing. The content image data DGc is data indicating a content image Gc indicating a content. The content image is an image indicating a photograph, a pattern, a color, or a combination of these as content. The content image may be a moving image or a still image. The content image data DGc may include at least one data indicating the content image Gc. In this embodiment, the content image data DGc includes a plurality of data files indicating the content image Gc. The method of acquiring the data file indicating the content image is arbitrary. For example, at least one of the data files may be stored in advance in the information processing device 30, or may be acquired from outside the information processing device 30. The editing image data DGd is data indicating an image in which the content image Gc indicated by the content image data DGc is superimposed on the line drawing indicated by the line drawing data DGb. However, before the generation of the line drawing data DGb, the editing image data DGd is data indicating an image in which the content image indicated by the content image data DGc is superimposed on the image indicated by the initial image data DGa0. The projection image data DGe is data indicating an image to be projected onto the object OJa by the projector 20. The conversion data DGf is data indicating a conversion matrix for performing a projective conversion between the coordinate system of the captured image of the camera 10 and the coordinate system of the display panel of the projector 20.The editing image data DGd may use the captured image data DGa instead of the line drawing data DGb. That is, the image represented by the editing image data DGd may be an image in which the content image represented by the content image data DGc is superimposed on the captured image represented by the captured image data DGa. However, when the editing image data DGd uses the line drawing data DGb, the content image can be suitably edited as described later in detail.

[0025] The processing device 32 is a processing device having a function of controlling each part of the information processing device 30 and a function of processing various data. The processing device 32 is configured to include a processor such as a CPU (Central Processing Unit). The processing device 32 may be configured with a single processor or multiple processors. In addition, some or all of the functions of the processing device 32 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0026] The communication device 33 is a communication device capable of communicating with the projector 20 and the like. For example, the communication device 33 is a wired communication device such as a wired LAN (Local Area Network), a Universal Serial Bus (USB), or an HDMI (High Definition Multimedia Interface), or a wireless communication device such as a LPWA (Low Power Wide Area), a wireless LAN including Wi-Fi, or Bluetooth. "HDMI", "Wi-Fi", and "Bluetooth" are each registered trademarks. The communication device 33 may be capable of communicating with the camera 10, or may be capable of communicating with an external device such as a server.

[0027] The display device 34 displays various images under the control of the processing device 32. The display device 34 is a display device including various display panels such as a liquid crystal display panel and an organic EL (electro-luminescence) display panel.

[0028] The input device 35 is an input device that accepts operations from a user. For example, the input device 35 includes a touch pad, a touch panel, or a pointing device such as a mouse. Here, when the input device 35 includes a touch panel, it may also serve as the display device 34.

[0029] In the above information processing device 30, the processing device 32 realizes various functions by referring to the above-mentioned various data stored in the storage device 31 and by executing the program PR stored in the storage device 31. Specifically, the processing device 32 functions as an acquisition unit 32a, a display control unit 32b, an image editing unit 32c, a projector control unit 32d, and a measurement unit 32e by executing the program PR. Therefore, the processing device 32 includes the acquisition unit 32a, the display control unit 32b, the image editing unit 32c, the projector control unit 32d, and the measurement unit 32e.

[0030] The acquisition unit 32a controls the operation of the communication device 33 to acquire various pieces of information from various devices connected to the information processing device 30, and stores the acquired information in the storage device 31. For example, the acquisition unit 32a acquires captured image data DGa from the camera 10, and acquires content image data DGc from a server (not shown) or the like.

[0031] The display control unit 32b controls the operation of the display device 34 to cause the display device 34 to display various information. Specifically, the display control unit 32b causes the display device 34 to display a user interface image GU required for executing a content editing method described below. Here, the display control unit 32b causes the display device 34 to display a line drawing indicated by the line drawing data DGb, and causes the display device 34 to display an editing image indicated by the editing image data DGd.

[0032] The image editing unit 32c executes various processes required for executing the content editing method described later in detail. Specifically, the image editing unit 32c generates line drawing data DGb based on the captured image data DGa, generates image data for editing DGd based on the line drawing data DGb and the content image data DGc, and edits the content image indicated by the image data for editing DGd based on the input result from the user. Here, the line drawing data DGb is generated, for example, by processing the captured image indicated by the captured image data DGa using a known line drawing generation technique. Therefore, the objects OJa, OJb, and OJc may be objects whose boundaries can be detected from the initial captured image Ga. In other words, the objects OJa, OJb, and OJc are not limited to objects independent of each other, and may be at least partially continuous or flat. The image data for editing DGd is generated, for example, by combining the line drawing data DGb or the initial image data DGa0 with the content image data DGc. The content image Gc indicated by the editing image data DGd is edited, for example, by editing the content image data DGc based on the input from the user, and then combining the line drawing data DGb or the preliminary image data DGa0 with the content image data DGc.

[0033] Furthermore, the image editing unit 32c generates the projection image data DGe by conversion using the conversion data DGf based on the edited content image data DGc. Note that the generation of the projection image data DGe may be performed by the projector control unit 32d.

[0034] The measurement unit 32e generates the conversion data DGf by measuring the projection surface of the projector 20 using the camera 10 and the projector 20. Specifically, the measurement unit 32e acquires a plurality of captured images by sequentially projecting a plurality of pattern images onto the projection target object using the projector 20 and capturing each of the projected pattern images using the camera 10. This results in the capture image data DGa. Then, based on the plurality of captured images, the measurement unit 32e generates a transformation matrix for performing projective transformation of these coordinate systems by associating the coordinates of the pattern images in the coordinate system of the captured image of the camera 10 with the coordinates of the pattern images in the coordinate system of the display panel of the projector 20. This results in the conversion data DGf.

[0035] The projector control unit 32d controls the operation of the projector 20 to cause the projector 20 to display various information. Specifically, the projector control unit 32d causes the projector 20 to project an image indicated by the projection image data DGe.

[0036] 1-3. How to edit content 3 is a flowchart showing the flow of the content editing method according to the first embodiment. The content editing method is executed by the information processing device 30 described above.

[0037] More specifically, as shown in FIG. 3, first, in step S101, the display control unit 32b causes the display device 34 to display the user interface image GU when the program PR is started.

[0038] Next, in step S102, the measurement unit 32e determines whether or not the installation of the projector 20 is complete. This determination is executed until it is determined that the installation of the projector 20 is complete based on an operation on the user interface image GU (step S102: NO).

[0039] When it is determined that the installation of the projector 20 is complete (step S102: YES), in step S103, the measurement unit 32e determines whether or not there is an instruction to start scanning for measuring the projection surface of the projector 20. This determination is executed based on an operation on the user interface image GU until there is an instruction to start scanning (step S103: NO).

[0040] If an instruction to start a scan has been given (step S103: YES), the measurement unit 32e starts a scan in step S104.

[0041] Next, in step S105, the display control unit 32b causes the display device 34 to display an initial image which is an image indicated by the initial image data DGa0.

[0042] Next, in step S106, the display control unit 32b causes the display device 34 to display a plurality of images as candidates for content images based on the content image data DGc.

[0043] Next, in step S107, the display control unit 32b determines whether or not a content image has been selected. This determination is repeated until a content image is selected based on an operation on the user interface image GU (step S107: NO).

[0044] If a content image has been selected (step S107: YES), in step S108, the display control unit 32b causes the display device 34 to display the selected image as a content image. At this time, the image editing unit 32c generates editing image data DGd based on data indicating the selected content image and the preliminary image data DGa0. Then, the display device 34 causes the image indicated by the editing image data DGd to be displayed on the display device 34. As a result, the selected content image is displayed on the display device 34 superimposed on the image indicated by the preliminary image data DGa0.

[0045] Next, in step S109, the display control unit 32b judges whether the scan has ended. This judgment is repeated until the scan has ended (step S109: NO). Thus, the scan is performed as the first process S1 over the period from step S104 to step S109 described above. During the execution of the first process S1, the second process S2 including the above-mentioned steps S105 to S108 is executed.

[0046] In this way, the program PR causes the computer to execute the first process S1 and the second process S2. Here, the first process S1 acquires conversion data DGf used to adjust the projection image projected onto the projection area RP based on a plurality of captured images obtained by capturing a plurality of pattern images sequentially projected onto the projection area RP from the projector 20. Also, the second process S2 is a process different from the first process S1 and is related to the projection image, and executes the second process S2 during the period in which the first process S1 is being executed.

[0047] When the scan is completed (step S109: YES), in step S110, the image editing unit 32c generates line drawing data DGb based on the captured image data DGa, and then the display control unit 32b causes the display device 34 to display an image indicated by the line drawing data DGb. At this time, the image editing unit 32c generates editing image data DGd based on the data indicating the selected content image and the line drawing data DGb, and updates the editing image data DGd. Then, the display device 34 causes the display device 34 to display an image indicated by the updated editing image data DGd. As a result, the selected content image is displayed on the display device 34 superimposed on the image indicated by the line drawing data DGb.

[0048] Next, in step S111, the image editing unit 32c judges whether or not there is an instruction for a cut shape for the content image. This judgment is repeated until there is an instruction for a cut shape for the content image (step S111: NO).

[0049] If an instruction for a cut shape of the content image is given (step S111: YES), in step S112, the image editing unit 32c edits the content image displayed on the display device 34 according to at least one of the position, shape, and size based on the instruction.

[0050] Next, in step S113, the image editing unit 32c determines whether or not to adjust the contour shape of the content image. This adjustment is performed using the user interface image GU.

[0051] If the contour shape of the content image has been adjusted (step S113: YES), in step S114, the image editing unit 32c edits the contour shape of the content image displayed on the display device 34 in accordance with the adjustment.

[0052] After that, or if there is no adjustment to the contour shape of the content image (step S113: NO), the image editing unit 32c determines in step S115 whether or not there is an instruction to project by the projector 20. If there is no instruction to project by the projector 20 (step S113: NO), the image editing unit 32c returns to the above-mentioned step S111.

[0053] If a command to project by the projector 20 has been issued (step S115: YES), in step S116, the image editing unit 32c generates projection image data DGe based on information on the position, shape, and size of the edited content image. Here, the projection image data DGe is generated by conversion using the conversion data DGf based on the edited content image data DGc.

[0054] The above is the flow of the content editing method. An example of a user interface image GU used in the content editing method will be described below with reference to Figs. 4 to 13. Note that Figs. 4 to 13 show user interface images GU-0 to GU-9 as user interface images GU that change according to the progress of content editing. Hereinafter, the user interface images GU-0 to GU-9 may be referred to as user interface images GU without distinguishing between them. Also, the user interface images GU are not limited to the examples shown in Figs. 4 to 13.

[0055] Fig. 4 is a diagram for explaining the installation of projector 20. Fig. 4 shows a user interface image GU-0 that is displayed on display device 34 when the above-mentioned step S102 is executed. User interface image GU-0 includes regions R1 and R2 and a button B0.

[0056] The progress of content editing is displayed in area R1. In the example shown in Fig. 4, the items "Installation", "Scan", "Content Creation", "Projection", and "Complete" are displayed in area R1, and items to be executed or executed are displayed in a distinguishable manner from items that have not been executed.

[0057] Here, the items "Installation," "Scan," "Content Creation," "Projection," and "Completion" are executed in this order. The "Installation" item is a display indicating that guidance regarding the installation of the camera 10 and projector 20 will be provided. The "Scan" item is a display indicating that shape measurement of the projection object will be performed. The "Content Creation" item is a display indicating that editing of a content image will be performed. The "Projection" item is a display indicating that projection will be performed by the projector 20. "Completion" is a display indicating that content editing has been completed.

[0058] Area R1 of user interface image GU-0 indicates that "Installation" is the execution target, and area R1 displays the text "Please shine the projector light on the target object" which shows how to install projector 20. At this time, camera 10 can capture the projection target object.

[0059] Necessary images are displayed in the region R2 according to the progress of the content editing. The captured image G0 by the camera 10 is displayed in the region R2 of the user interface image GU-0. The captured image G0 includes an image G0a showing an object OJa, an image G0b showing an object OJb, and an image G0c showing an object OJc. This allows the user to visually confirm in the region R2 that the projection objects are present in both the captured region RC of the camera 10 and the projection region RP of the projector 20.

[0060] The button B0 is a display for accepting the advancement of the execution target item to the next item when the installation of the projector 20 is completed, and is used for the judgment of the above-mentioned step S102. When the button B0 is operated, the measurement unit 32e judges that the installation is completed (step S102: YES). Then, the execution target item becomes "scan", and the above-mentioned step S103 is executed.

[0061] Fig. 5 is a diagram for explaining an instruction to start execution of the first process S1. Fig. 5 shows a user interface image GU-1 that is displayed on the display device 34 when the above-mentioned step S103 is executed. The user interface image GU-1 includes regions R1 and R2 and buttons B1 and B2.

[0062] Area R1 of user interface image GU-1 indicates that "scan" is to be executed, and area R1 displays the text "Scan will run for 5 minutes" to prompt the user to start "scan." Area R1 also displays the text "You can operate the next step while scanning," indicating that the second process S2 can be executed while the first process S1 is being executed. This allows the user to know that a process other than "scan" can be executed while "scan" is being executed.

[0063] In the area R2 of the user interface image GU-1, the captured image G0 by the camera 10 is displayed.

[0064] The button B1 is a display for accepting a request to return the item to be executed to the previous item. When the button B1 is operated, the item to be executed becomes "Install."

[0065] Button B2 is an indication for accepting the start of processing of the item to be executed. When button B2 is operated, the measurement unit 32e determines that an instruction to start scanning has been given (step S103: YES). Then, "scanning", which is the item to be executed, is started. Furthermore, when "scanning" is started, the item to be executed transitions to "content creation".

[0066] When the execution of "scanning" is started, the measurement unit 32e controls the operation of the projector 20 so as to sequentially project a plurality of pattern images onto the projection object, and controls the operation of the camera 10 so as to capture each pattern image projected onto the projection object. In this way, a plurality of captured images of the pattern images captured by the camera 10 are obtained. Then, the measurement unit 32e measures the projection surface, which is the surface of the projection object, based on the plurality of captured images. Measuring the projection surface means generating a transformation matrix for performing projective transformation between the coordinate system of the captured image of the camera 10 and the coordinate system of the display device of the projector 20. The measurement unit 32e generates the transformation matrix by associating the coordinates of the pattern image in the coordinate system of the captured image of the camera 10 with the coordinates of the pattern image in the coordinate system of the display device of the projector 20 based on the plurality of captured images.

[0067] As the pattern image, for example, a binary code pattern is used. The binary code pattern is an image for expressing the coordinates of a display device using a binary code. The binary code is a technique for expressing the value of each digit when an arbitrary numerical value is expressed in binary by turning a switch on and off. When a binary code pattern is used as the pattern image, the image projected by the projector 20 corresponds to the switch, and pattern images are required for the number of digits of the binary number expressing the coordinate value. In addition, separate pattern images are required for each of the vertical coordinate and the horizontal coordinate. For example, when the resolution of the display panel of the projector 20 is 120×90, 120 and 90 are each expressed by a 7-digit binary number, so seven images are required to express the vertical coordinate and seven images are required to express the horizontal coordinate. Here, the multiple pattern images include one or both of a white image that displays white color over the entire projection area RP and a black image that displays black color over the entire projection area RP.

[0068] Furthermore, when a binary code pattern is used as a pattern image, the robustness of the measurement generally decreases due to the influence of disturbance light such as lighting. Therefore, when a binary code pattern is used as a pattern image, it is preferable to use a complementary pattern in combination with the binary code pattern in order to reduce the influence of disturbance light and improve the robustness of the measurement. A complementary pattern is an image in which black and white are inverted.

[0069] It should be noted that the pattern image is not limited to a binary code pattern, and may be other structured light such as a dot pattern, a rectangular pattern, a polygonal pattern, a checkered pattern, a gray code pattern, a phase shift pattern, or a random dot pattern.

[0070] Fig. 6 is a diagram for explaining the selection of a content image. Fig. 6 shows a user interface image GU-2 that is displayed on the display device 34 when the above-mentioned step S107 is executed. User interface image GU-2 includes buttons B3 and B4 instead of buttons B1 and B2 of user interface image GU-1, and also includes an area R3.

[0071] In the region R3, an image Gcg including a plurality of content images indicated by the content image data DGc is displayed as a candidate. Each of the plurality of content images is capable of accepting selection. The above-mentioned step S107 is judged depending on the presence or absence of this selection. When this selection is made, the display control unit 32b judges that a content image has been selected (step S107: YES). Then, the above-mentioned step S108 is executed. Note that the number of content images displayed in the region R3 is not limited to the example shown in FIG. 6, and is arbitrary, and may be three or less, five or more, or may be a single image.

[0072] In this way, when the program PR receives an input from the user in step S107 during the execution of the first process S1, it executes step S108 of the second process S2. The second process S2 also includes selecting a content image Gc to be used for at least a part of the projection image in step S107.

[0073] The program PR of the present embodiment causes a plurality of candidates for the content image Gc to be displayed in step S106. The second process S2 then includes selecting one of the plurality of candidates as the content image Gc in step S107.

[0074] Region R1 of user interface image GU-2 indicates that "content creation" is the execution target, and region R1 displays the text "Please select the content to project" to prompt the user to select a content image. Region R1 of user interface image GU-2 also displays the progress of the scan. Figure 6 illustrates an example in which the scan progress is 10%.

[0075] In the region R2 of the user interface image GU-2, the initial captured image Ga indicated by the initial image data DGa0 is displayed. The initial captured image Ga includes an image Gaa indicating the object OJa. The initial image data DGa0 is generated during scanning. After the initial image data DGa0 is generated, the initial captured image Ga indicated by the initial image data DGa0 is displayed in step S105.

[0076] Thus, the second process S2 includes, in step S105, a process using the initial image data DGa0, which is an example of information obtained during execution of the first process S1. Here, the captured images indicated by the captured image data DGa include an initial captured image Ga, which is a captured image of the projection area RP in an initial state. And, the second process S2 includes, in step S105, a process using the initial captured image Ga.

[0077] Button B3 is an indication for accepting that the item to be executed is returned to "Scan," the item before "Content Creation." Button B4 is an indication for accepting that the selection of the content image has been completed. When button B4 is operated, the above-mentioned step S111 is executed. However, operation of button B4 is permitted after step S110 has been executed, and cannot be accepted otherwise.

[0078] Fig. 7 is a diagram showing a display example of a content image Gc superimposed on an initial captured image Ga. Fig. 7 shows a user interface image GU-3 that is displayed on the display device 34 by executing the above-mentioned step S108. The user interface image GU-3 is displayed when one content image is selected in the user interface image GU-2.

[0079] Fig. 7 shows a state in which the lower left content image is selected among the four content images displayed in region R3. Fig. 7 also shows an example in which text indicating that scanning is complete is displayed in region R1. When scanning is complete, step S110 is executed, and operation on button B4 becomes available.

[0080] In the region R2 of the user interface image GU-3, an editing image Gd indicated by the editing image data DGd is displayed. Here, the editing image Gd is an image in which the initial captured image Ga indicated by the initial image data DGa0 and the content image Gc indicated by the content image data DGc are superimposed. In the example shown in Fig. 7, the content image Gc indicated by the content image data DGc is superimposed on a part of the initial captured image Ga. Note that the content image Gc only needs to be superimposed on at least a part of the initial captured image Ga, and may be superimposed on the entire initial captured image Ga.

[0081] In this way, the second process S2 includes a process of displaying the content image Gc superimposed on the initial captured image Ga in step S108.

[0082] 8 and 9 are diagrams showing examples of displays after the first process S1. FIG. 8 shows a user interface image GU-4 that is displayed on the display device 34 when the above-mentioned step S111 is executed. FIG. 9 shows a user interface image GU-5 that is displayed on the display device 34 when the above-mentioned step S111 is executed. The user interface images GU-4 and GU-5 include buttons B5 and B6 instead of buttons B3 and B4 of the user interface image GU-3. Note that the display of the region R3 is omitted in the user interface images GU-4 and GU-5.

[0083] Area R1 of user interface image GU-4 indicates that “content creation” is the target to be executed, and after selecting a content image, area R1 displays text such as “Tap to select the shape to cut,” prompting the user to specify the cut shape of the content image.

[0084] A line drawing Gb represented by the line drawing data DGb is displayed in region R2 of the user interface image GU-4. The line drawing Gb includes an image Gba representing an object OJa, etc. This can improve the visibility of the contour of the projection target. Here, the image Gba is represented as a closed region partitioned by the lines of the line drawing Gb.

[0085] In this way, after the execution of the first process S1 ends, the program PR causes, in step S110, the image Gba showing the outline of the object OJa included in the projection region RP by lines to be displayed in place of the initial captured image Ga.

[0086] Here, the line drawing Gb can accept the selection of a closed area partitioned by the lines of the line drawing Gb. When this selection is made, the shape of the selected closed area is designated as the cut shape, and the above-mentioned step S112 is executed. In the example shown in FIG. 8 and FIG. 9, a desired closed area is selected by tapping the cursor CUR. FIG. 8 shows a state before the closed area corresponding to the image Gba showing the object OJa is selected. FIG. 9 shows a state in which the closed area corresponding to the image Gba showing the object OJa is selected.

[0087] Here, the selected closed region is highlighted. In the example shown in Fig. 9, the inside of the selected closed region is displayed in a color different from other regions. Note that the highlighting method is not limited to the example shown in Fig. 9, and may be, for example, a method of thickening the line showing the outline of the selected closed region.

[0088] The button B5 is a display for accepting that the content image is returned to a selectable state. The button B6 is a display for accepting that the cut shape designation has been completed. When the button B6 is operated, the image editing unit 32c determines that a cut shape designation for the content image has been made (step S111: YES). Then, the above-mentioned step S112 is executed.

[0089] Fig. 10 is a diagram for explaining editing of the content image after the first process S1. Fig. 10 shows a user interface image GU-6 displayed on the display device 34 when the above-mentioned step S113 is executed. The user interface image GU-6 is similar to the above-mentioned user interface images GU-4 and GU-5, except that it includes buttons B7 and B8 instead of buttons B5 and B6.

[0090] Area R1 of user interface image GU-6 indicates that "content creation" is the target to be executed, and area R1 displays text such as "You can adjust the shape of the content," indicating that the shape of the content image can be adjusted.

[0091] In region R2 of user interface image GU-6, the content image Gc, which has been trimmed to fit the shape of image Gba, is displayed superimposed on the line drawing Gb. Note that the content image Gc in the trimmed shape may be displayed by editing the content image data DGc so as to cut out unnecessary parts of the content image Gc, or may be displayed by displaying a mask image that hides unnecessary parts of the content image Gc without editing the content image data DGc.

[0092] Here, the content image Gc can receive an adjustment of the contour shape. In the example shown in FIG. 10, a plurality of dots are arranged along the contour of the content image Gc. The adjustment of the contour shape of the content image Gc is performed by using the cursor CUR to select any one dot from the plurality of dots, and then moving the selected dot. When this adjustment is performed, the image editing unit 32c determines that the contour shape of the content image has been adjusted (step S113: YES). Then, the above-mentioned step S114 is executed. Note that the display for adjusting the contour shape of the content image Gc is not limited to the example shown in FIG. 10, and is arbitrary.

[0093] Button B7 is a display for accepting a return to a state in which the cut shape can be specified. Button B8 is a display for accepting a move to the next item as the execution target. When button B8 is operated, the execution target shifts to "Projection".

[0094] Fig. 11 is a diagram for explaining generation of a projection image. Fig. 11 shows a user interface image GU-7 displayed on the display device 34 during execution of the above-mentioned step S115. The user interface image GU-7 is similar to the above-mentioned user interface image GU-6, except that it includes buttons B9 and B10 instead of buttons B7 and B8, and that the initial captured image Ga is displayed in the area R2 instead of the line drawing Gb.

[0095] Area R1 of user interface image GU-7 indicates that "projection" is the target to be executed, and area R1 displays text such as "This content will be projected" indicating that it is possible to project a content image.

[0096] In region R2 of the user interface image GU-7, the trimmed content image Gc is displayed superimposed on the initial captured image Ga. In this way, by superimposing the trimmed content image Gc on the initial captured image Ga, an image close to the actual projection state of the content image Gc can be visually provided by the display device 34. Note that the captured image displayed in the user interface image GU-7 does not have to be the initial captured image Ga. For example, it may be an image newly captured after the completion of "scanning."

[0097] The button B9 is a display for accepting a change of the item to be executed back to the previous item. When the button B9 is operated, the item to be executed becomes "Create content."

[0098] The button B10 is an indication for accepting the start of a generation process of the projection image data DGe. When the button B10 is operated, the image editing unit 32c determines that an instruction for projection by the projector 20 has been given (step S115: YES). Then, by executing the above-mentioned step S116, the projection image data DGe is generated based on the edited content image Gc using the conversion data DGf.

[0099] In this manner, in step S116, the program PR causes the content image Gc to be edited using the conversion data DGf, thereby generating a projection image.

[0100] Fig. 12 is a diagram showing an example of a display during generation of a projection image. Fig. 12 shows a user interface image GU-8 displayed on the display device 34 during execution of the above-mentioned step S116. User interface image GU-8 is similar to the above-mentioned user interface image GU-7, except that buttons B9 and B10 are omitted.

[0101] Area R1 of user interface image GU-8 indicates that "projection" is the target to be executed, and area R1 displays the text "Adjusting. Please wait.", indicating that the generation process of projection image data DGe is currently underway.

[0102] An image indicating that processing is in progress is displayed in the region R2 of the user interface image GU-8. Note that the image is not limited to the example shown in FIG.

[0103] Fig. 13 is a diagram showing a display example when generation of a projection image is completed. Fig. 13 shows a user interface image GU-9 displayed on the display device 34 after execution of the above-mentioned step S116. User interface image GU-9 is similar to the above-mentioned user interface image GU-7, except that it includes buttons B11, B12, and B13 instead of buttons B9 and B10.

[0104] Area R1 of user interface image GU-9 indicates that "Complete" is the action to be taken, and area R1 displays text such as "Adjustment of projected content is complete" indicating that content editing is complete.

[0105] The button B11 is a display for accepting a request to return the item to be executed to the previous item. When the button B11 is operated, the item to be executed becomes "Create content."

[0106] The button B12 is a display for accepting the end of content editing by the user interface image GU. When the button B12 is operated, the display of the user interface image GU is ended. Projection image data DGe indicating the generated projection image is stored in the storage device 31.

[0107] The button B13 is a display for accepting the return of the execution target item to "Install." When the button B13 is operated, the execution target item becomes "Install."

[0108] Fig. 14 is a diagram for explaining projection using the edited content image Gc. The information processing device 30 reads out the projection image data DGe from the storage device 31 and outputs it to the projector 20 via the communication device 33. As a result, the edited content image Gc as described above is projected onto the object OJa by the projector 20, as shown in Fig. 14.

[0109] As described above, the content editing method includes executing a first process S1 to obtain conversion data DGf used to adjust the projection image projected onto the projection area RP based on a plurality of captured images each of which is a plurality of pattern images projected sequentially from the projector 20 onto the projection area RP, and executing, during the period in which the first process S1 is being executed, a second process S2 relating to the projection image, which is a process different from the first process S1.

[0110] As described above, the information processing device 30 includes the processing device 32 that executes the above-mentioned content editing method. The processing device 32 executes a first process S1 for acquiring conversion data DGf used to adjust a projection image projected onto the projection area RP, based on a plurality of captured images obtained by capturing a plurality of pattern images sequentially projected onto the projection area RP from the projector 20. During the period in which the first process S1 is being executed, the processing device 32 executes a second process S2 related to the projection image, which is different from the first process S1.

[0111] In the above content editing method, information processing device 30, or program PR, the second process S2, which is different from the first process S1 and is related to the projection image, is executed during the period in which the first process S1 is being executed, so that the time required from the start of the first process S1 to the completion of the second process S2 can be shortened compared to an aspect in which the second process S2 is executed after the completion of execution of the first process S1. In other words, by effectively utilizing the period during which the first process S1 is being executed to execute the second process S2, which is different from the first process S1, the time required from the start of the first process S1 to the completion of the second process S2 can be shortened.

[0112] As described above, the program PR of this embodiment executes the second process S2 when an input from the user is received during the execution of the first process S1. This makes it possible to effectively use the execution period of the first process S1 for the execution of the second process S2 according to the user's intention.

[0113] As described above, the second process S2 includes selecting a content image Gc to be used for at least a part of the projection image. Therefore, the execution period of the first process S1 can be effectively used for selecting the content image Gc. Here, the selection of the content image Gc can be performed during the execution period of the first process S1 because the final processing result of the first process S1 is not used.

[0114] Furthermore, as described above, the program PR further causes the display of multiple candidates for the content image Gc. The second process S2 includes selecting one of the multiple candidates as the content image Gc. Therefore, the execution period of the first process S1 can be effectively used for displaying the multiple candidates for the content image. Here, the display of the multiple candidates can be executed during the execution period of the first process S1 because the final processing result of the first process S1 is not used.

[0115] As described above, the second process S2 includes a process using information obtained during the execution of the first process S1. Therefore, the execution period of the first process S1 can be effectively used for the process using information obtained during the execution of the first process S1. Here, the process using information obtained during the execution of the first process S1 does not use the final processing result of the first process S1, and therefore can be executed during the execution period of the first process S1.

[0116] Furthermore, as described above, the multiple captured images represented by the captured image data DGa include an initial captured image Ga, which is an image captured of the projection area RP in an initial state. The second process S2 includes a process using the initial captured image Ga. Therefore, the execution period of the first process S1 can be effectively utilized for the process using the initial captured image Ga. Here, the process using the initial captured image Ga does not use the final processing result of the first process S1, and therefore can be executed during the execution period of the first process S1.

[0117] As described above, the second process S2 includes a process of displaying the content image Gc superimposed on the initial captured image Ga. Therefore, the execution period of the first process S1 can be effectively utilized for the process of displaying the content image Gc superimposed on the initial captured image Ga. Here, the process of displaying the content image Gc superimposed on the initial captured image Ga does not use the final processing result of the first process S1, and therefore can be executed during the execution period of the first process S1.

[0118] Furthermore, as described above, the projection region RP includes the object OJa. After the first process S1 is completed, the program PR further executes displaying an image Gba showing the outline of the object OJa in line instead of the initial captured image Ga. Therefore, the final processing result of the first process S1 can be used to display the image Gba showing the outline of the object in line with high accuracy.

[0119] As described above, the program PR further executes the process of generating a projection image by editing the content image Gc using the conversion data DGf. Therefore, after the first process S1 is completed, a highly accurate projection image can be generated by editing the content image Gc using the conversion data DGf.

[0120] 2. Second embodiment A second embodiment of the present disclosure will be described below. In the following exemplary embodiment, for elements whose actions and functions are similar to those of the first embodiment, the reference symbols used in the description of the first embodiment will be used and detailed descriptions of each will be omitted as appropriate.

[0121] 15 is a flowchart showing the flow of a content editing method according to the second embodiment. This embodiment is similar to the first embodiment described above, except that steps S117 and S118 are executed instead of steps S109 and S110.

[0122] In this embodiment, as shown in FIG. 15, after step S108 is executed, in step S117, the image editing unit 32c generates line drawing data DGb based on the captured image data DGa, and then the display control unit 32b displays an image represented by the line drawing data DGb on the display device 34. In step S117, the scan is incomplete. Captured images have been acquired for multiple pattern images, but capturing of all pattern images has not been completed. Therefore, in step S117, a transformation matrix is ​​generated from the captured captured images. The accuracy of the transformation matrix is ​​lower than when all pattern images are used.

[0123] Next, in step S118, the display control unit 32b determines whether or not an instruction to end the scan has been issued. This determination is made based on an operation on the user interface image GU.

[0124] If there is no instruction to end the scan (step S118: NO), the scan continues and the above-mentioned step S117 is executed. On the other hand, if there is an instruction to end the scan (step S118: YES), the scan is stopped and step S111 is executed. In the processes from step S111 onwards, a transformation matrix generated based on the captured images captured up until the instruction to end the scan is given is used. Note that if there is no instruction to end the scan by the time the scan is completed, the process proceeds to step S111 when the scan is completed.

[0125] In this manner, in this embodiment, scanning is executed as the first process S1 over the period from step S104 to step S118 described above. During the execution of the first process S1, the second process S2 including steps S105 to S117 described above is executed.

[0126] Fig. 16 is a diagram showing a display example of a content image Gc superimposed on a line drawing Gb. Fig. 16 shows a user interface image GU-3 displayed on the display device 34 by executing the above-mentioned step S117. The user interface image GU-3 of this embodiment is similar to the user interface image GU-3 of the first embodiment, except that the line drawing Gb indicated by the line drawing data DGb is displayed in the area R2 instead of the initial captured image Ga.

[0127] That is, in the region R2 of the user interface image GU-3 of the present embodiment, an editing image Gd in which the line drawing Gb indicated by the line drawing data DGb and the content image Gc indicated by the content image data DGc are superimposed is displayed. In the example shown in Fig. 16, the content image Gc indicated by the content image data DGc is superimposed on a part of the line drawing Gb.

[0128] The above second embodiment also allows the processing time required for editing the content to be shortened. In this embodiment, as described above, the second process S2 includes displaying an image Gba showing the outline of the object OJa included in the projection area RP in place of the initial captured image Ga, and accepting an input to stop the execution of the first process S1. Therefore, during the execution period of the first process S1, the image Gba showing the outline of the object OJa in lines can be displayed with a display accuracy according to the progress of the first process S1, using information obtained during the execution of the first process S1. In addition, the first process S1 can be stopped at a stage where the display accuracy required by the user is obtained. Therefore, the time required from the start of the first process S1 to the completion of the second process S2 can be shortened according to the user's desire.

[0129] 3. Variations Each of the above-mentioned exemplary embodiments may be modified in various ways. Specific modified embodiments that may be applied to each of the above-mentioned embodiments are illustrated below. Two or more embodiments selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0130] 3-1. Variation 1 In the above-described embodiment, an example is given of a form in which the cut shape of a content image is specified using an image showing the object OJa in line drawing, but this is not limited to this form, and for example, the image for specifying the cut shape of a content image may be an image that shows the contour of the object OJa by using different colors inside and outside the contour.

[0131] 3-2. Variation 2 In the above embodiment, the number of objects that can be the projection target is one, but the present invention is not limited to this, and the number may be two or more.

[0132] 3-3. Variation 3 In the above embodiment, the second process is exemplified by selecting a content image, but is not limited to this. For example, the process related to generation of a projection image may be a process of adding image data to the content image data DGc. Specifically, image data stored separately from the content image data DGc in the storage device 31, such as a photo or a video, may be displayed on the display device 34 as a candidate to be added to the content image data DGc, and may be added to the content image data DGc in response to a user's selection operation. Candidates for images to be added may be obtained from an external device accessible by the information processing device 30 via the communication device 33.

[0133] 4. Notes The following is a summary of this disclosure.

[0134] (Additional Note 1) A program that causes a computer to execute a first process that acquires measurement information used to adjust a projection image projected onto a projection area based on a plurality of captured images that capture a plurality of pattern images that are sequentially projected onto a projection area from a projector, and executes a second process related to the projection image, which is different from the first process, during the period in which the first process is being executed.

[0135] In the aspect of Supplementary Note 1 above, since the second process, which is different from the first process and is related to the projection image, is executed during the period in which the first process is being executed, the time required from the start of the first process to the completion of the second process can be shortened compared to an aspect in which the second process is executed after the completion of the execution of the first process. In other words, by effectively utilizing the period in which the first process is being executed to execute the second process, which is different from the first process, the time required from the start of the first process to the completion of the second process can be shortened.

[0136] (Supplementary Note 2) The program of Supplementary Note 1, which executes the second process when an input from a user is received during the execution of the first process. In the above aspect of Supplementary Note 2, the execution period of the first process can be effectively used for the execution of the second process according to the user's intention.

[0137] (Supplementary Note 3) The program of Supplementary Note 1 or Supplementary Note 2, wherein the second process includes selecting a content image to be used for at least a part of the projection image. In the aspect of Supplementary Note 3 above, the execution period of the first process can be effectively used for the selection of the content image. Here, the selection of the content image can be performed during the execution period of the first process because the final processing result of the first process is not used.

[0138] (Supplementary Note 4) The program of Supplementary Note 3, further comprising: displaying a plurality of candidates for a content image, and the second process includes selecting one of the plurality of candidates as the content image. In the above aspect of Supplementary Note 4, the execution period of the first process can be effectively utilized for displaying a plurality of candidates for a content image. Here, the display of the plurality of candidates can be performed during the execution period of the first process, since the final processing result of the first process is not used.

[0139] (Supplementary Note 5) The program according to any one of Supplementary Note 1 to Supplementary Note 4, wherein the second process includes a process using information obtained during execution of the first process. In the aspect of Supplementary Note 5 above, the execution period of the first process can be effectively utilized for the process using information obtained during execution of the first process. Here, the process using information obtained during execution of the first process does not use the final processing result of the first process, and therefore can be executed during the execution period of the first process.

[0140] (Supplementary Note 6) The program of Supplementary Note 5, wherein the plurality of captured images include an initial captured image that is an image captured of the projection area in an initial state, and the second process includes a process using the initial captured image. In the above aspect of Supplementary Note 6, the execution period of the first process can be effectively utilized for the process using the initial captured image. Here, the process using the initial captured image does not use the final processing result of the first process, and therefore can be executed during the execution period of the first process.

[0141] (Supplementary Note 7) The program of Supplementary Note 6, wherein the second process includes a process of displaying a content image superimposed on the initial captured image. In the aspect of Supplementary Note 7 above, the execution period of the first process can be effectively utilized for the process of displaying a content image superimposed on the initial captured image. Here, the process of displaying a content image superimposed on the initial captured image can be executed during the execution period of the first process because the final processing result of the first process is not used.

[0142] (Supplementary Note 8) The program of Supplementary Note 7, further comprising: displaying, after the first process is completed, an image showing the outline of the object in lines instead of the initial captured image, wherein the projection area includes an object. In the above aspect of Supplementary Note 8, the final processing result of the first process can be used to display the image showing the outline of the object in lines with high accuracy.

[0143] (Supplementary Note 9) The program of Supplementary Note 7, wherein the projection area includes an object, and the second process further includes displaying an image showing the contour of the object in lines instead of the initial captured image, and receiving an input to stop the execution of the first process. In the above aspect of Supplementary Note 9, during the execution of the first process, an image showing the contour of the object in lines can be displayed using information obtained during the execution of the first process. Also, the first process can be stopped at a stage where the display accuracy required by the user is obtained. Therefore, the time required from the start of the first process to the completion of the second process can be shortened according to the user's desire.

[0144] (Supplementary Note 10) The program according to any one of Supplementary Note 1 to Supplementary Note 9, further comprising: generating the projection image by editing a content image using the measurement information. In the above aspect of Supplementary Note 10, after the first process is completed, a highly accurate projection image can be generated by editing the content image using the measurement information.

[0145] (Appendix 11) A content editing method comprising: executing a first process to obtain measurement information used to adjust a projection image projected onto a projection area based on a plurality of captured images each capturing a plurality of pattern images projected sequentially from a projector onto the projection area; and executing, during a period in which the first process is being executed, a second process related to the projection image, the second process being different from the first process.

[0146] In the aspect of Supplementary Note 11 above, since the second process, which is different from the first process and is related to the projection image, is executed during the period in which the first process is being executed, the time required from the start of the first process to the completion of the second process can be shortened compared to an aspect in which the second process is executed after the completion of the execution of the first process. In other words, by effectively utilizing the period in which the first process is being executed to execute the second process different from the first process, the time required from the start of the first process to the completion of the second process can be shortened.

[0147] (Addendum 12) An information processing device including a processing device that executes a first process to obtain measurement information used to adjust a projection image projected onto a projection area based on a plurality of captured images of a plurality of pattern images that are sequentially projected onto a projection area from a projector, and executes a second process related to the projection image, which is different from the first process, during a period in which the first process is being executed.

[0148] In the aspect of Supplementary Note 12 above, since the second process, which is different from the first process and is related to the projection image, is executed during the period in which the first process is being executed, the time required from the start of the first process to the completion of the second process can be shortened compared to the aspect in which the second process is executed after the completion of the execution of the first process. In other words, by effectively utilizing the period in which the first process is being executed to execute the second process, which is different from the first process, the time required from the start of the first process to the completion of the second process can be shortened. [Explanation of symbols]

[0149] 10...camera, 20...projector, 30...information processing device, 31...storage device, 32...processing device, 32a...acquisition unit, 32b...display control unit, 32c...image editing unit, 32d...projector control unit, 32e...measurement unit, 33...communication device, 34...display device, 35...input device, 100...system, B0...button, B1...button, B10...button, B11...button, B12...button, B13...button, B2...button, B3...button, B4...button, B5...button, B6...button, B7...button, B8...button , B9...button, CUR...cursor, DGa...captured image data, DGa0...initial image data, DGb...line drawing data, DGc...content image data, DGd...image data for editing, DGe...image data for projection, DGf...conversion data, G0...captured image, G0a...image, G0b...image, G0c...image, GU...user interface image, GU-0...user interface image, GU-1...user interface image, GU-2...user interface image, GU-3...user interface image, GU-4...user interface image, GU-5...user interface image, GU-6...user interface image, GU-7...user interface image, GU-8...user interface image, GU-9...user interface image, Ga...initial captured image, Gaa...image, Gb...line drawing, Gba...image, Gc...content image, Gd...image for editing, OJa...object, OJb...object, OJc...object, PR...program, R1...area, R2...area, R 3...area, RC...imaging area, RP...projection area, S1...first processing, S2...second processing, S101...step, S102...step, S103...step, S104...step, S105...step, S106...step, S107...step, S108...step, S109...step, S110...step, S111...step, S112...step, S113...step, S114...step, S115...step, S116...step, S117...step, S118...step, W...wall surface.

Claims

1. executing a first process for acquiring measurement information used for adjusting a projection image projected onto a projection area based on a plurality of captured images obtained by capturing a plurality of pattern images sequentially projected onto the projection area from a projector; executing a second process related to the projection image, the second process being different from the first process, during a period in which the first process is being executed; Have the computer execute program.

2. when an input from a user is received during the execution of the first process, the second process is executed. The program according to claim 1.

3. The second process includes selecting a content image to be used for at least a portion of the projection image. The program according to claim 1 or 2.

4. [0023] Further performing the steps of: displaying a plurality of candidates for the content image; The second process includes selecting one candidate from the plurality of candidates as the content image. The program according to claim 1.

5. The second process includes a process using information obtained during the execution of the first process. The program according to claim 1 or 2.

6. the plurality of captured images include an initial captured image which is a captured image of the projection area in an initial state, The second processing includes processing using the initial captured image. The program according to claim 5.

7. the second processing includes processing for displaying a content image by superimposing it on the initial captured image, The program according to claim 6.

8. the projection area includes an object; after completion of execution of the first process, displaying an image showing an outline of the object in place of the initial captured image; Further execute The program according to claim 7.

9. the projection area includes an object; The second process includes: displaying an image showing the outline of the object in place of the initial captured image; receiving an input to stop execution of the first process; Further comprising: The program according to claim 7.

10. generating the projection image by editing a content image using the measurement information; Further execute The program according to claim 1 or 2.

11. executing a first process for acquiring measurement information used for adjusting a projection image projected onto a projection area based on a plurality of captured images obtained by capturing a plurality of pattern images sequentially projected onto the projection area from a projector; executing a second process related to the projection image, the second process being different from the first process, during a period in which the first process is being executed; Including, How to edit content.

12. executing a first process for acquiring measurement information used for adjusting a projection image projected onto a projection area based on a plurality of captured images obtained by capturing a plurality of pattern images sequentially projected onto the projection area from a projector; executing a second process related to the projection image, the second process being different from the first process, during a period in which the first process is being executed; a processor for executing Information processing device.

Images