Projection method, projector, and program

Abstract

To optimize the number of projected structured lights even when there is structured light that is not normally captured due to disturbance or the like.SOLUTION: A projection method includes acquiring a plurality of pieces of imaging data by capturing images of M (M is a natural number greater than or equal to 2) pieces of structured light projected from a projector onto a projection target using a camera, determining, on the basis of the plurality of imaging data, whether there is at least one first structured light among the M pieces of structured light that has not been captured correctly, and, when it is determined that there is at least one first structured light, projecting N (N is a natural number smaller than M) pieces of structured light including at least one first structured light onto the projection target.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a projection method, a projector, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology for identifying the correspondence between multiple pixels of an image projected from a projection device and multiple pixels of an image captured by an imaging device, using a gray code image, which is an example of structured light. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-174999 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 does not take into consideration optimizing the number of structured light beams to be projected in the event that the imaging device is unable to properly capture the structured light beams due to disturbances such as ambient light while the structured light beams are being projected onto the display surface. [Means for solving the problem]

[0005] A projection method according to one embodiment of the present disclosure includes: acquiring a plurality of pieces of imaging data by capturing images of M (M is a natural number greater than or equal to 2) pieces of structured light projected from a projector onto a projection target using a camera; determining, based on the plurality of imaging data, whether or not there is at least one first structured light among the M pieces of structured light that has not been properly captured; and, if it is determined that there is at least one first structured light, projecting N (N is a natural number smaller than M) pieces of structured light including the at least one first structured light onto the projection target.

[0006] A projection method according to another aspect of the present disclosure includes: acquiring a plurality of pieces of imaging data by capturing images of M (M is a natural number greater than or equal to 2) pieces of structured light projected from a projector onto a projection target, with a camera; determining, based on the plurality of imaging data, whether or not at least one first structured light among the M pieces of structured light is not properly captured; determining, based on change information indicating a change in one or both of a position and an attitude of the projector relative to the projection target, whether or not the position and the attitude have changed; projecting N (N is a natural number less than M) pieces of structured light including the at least one first structured light onto the projection target when it is determined that the at least one first structured light is present and that one or both of the position and the attitude have not changed; and projecting the M pieces of structured light onto the projection target when it is determined that the at least one first structured light is present and that one or both of the position and the attitude have changed.

[0007] A system according to one aspect of the present disclosure includes an optical device and a processing device that controls the operation of the optical device, wherein the processing device performs the following operations: acquiring a plurality of pieces of imaging data in which M (M is a natural number greater than or equal to 2) pieces of structured light projected from the optical device onto a projection target are each captured by a camera; determining, based on the plurality of imaging data, whether or not there is at least one first structured light among the M pieces of structured light that has not been captured correctly; and, if it is determined that there is at least one first structured light, projecting N (N is a natural number smaller than M) pieces of structured light including the at least one first structured light onto the projection target.

[0008] A program according to one embodiment of the present disclosure causes a computer to perform the following steps: acquire a plurality of pieces of imaging data in which M (M is a natural number greater than or equal to 2) pieces of structured light projected from a projector onto a projection target are each captured by a camera; determine, based on the plurality of pieces of imaging data, whether or not there is at least one first structured light among the M pieces of structured light that has not been captured correctly; and, if it is determined that there is at least one first structured light, cause the projector to project N (N is a natural number smaller than M) pieces of structured light including the at least one first structured light onto the projection target. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing an outline of a system used in a projection method according to a first embodiment. [Figure 2] FIG. 1 is a block diagram of a projector used in a projection method according to a first embodiment. [Figure 3] 4 is a flowchart showing the flow of a projection method according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating a user interface image. [Figure 5] FIG. 10 is a diagram for explaining a first pattern. [Figure 6] 10A and 10B are diagrams for explaining abnormality detection when a Gray code pattern is used as structured light. [Figure 7] 10A and 10B are diagrams for explaining detection of anomalies when a phase shift pattern is used as structured light. [Figure 8] 10A and 10B are diagrams for explaining detection of anomalies when a phase shift pattern is used as structured light. [Figure 9] FIG. 10 is a diagram illustrating an example of a message. [Figure 10] FIG. 10 is a block diagram of a projector used in a projection method according to a second embodiment. [Figure 11] 10 is a flowchart showing the flow of a projection method according to a second embodiment. [Figure 12]FIG. 10 is a block diagram of a projector used in a projection method according to a third embodiment. [Figure 13] 10 is a flowchart showing the flow of a projection method according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described below with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions and are shown schematically to facilitate understanding. Furthermore, the scope of the present disclosure is not limited to these embodiments unless otherwise specified in the following description to the effect that the present disclosure is limited.

[0011] 1. First embodiment 1-1. System Overview 1 is a diagram showing an outline of a system 100 used in a projection method according to the first embodiment. The system 100 is a projection system that projects a projection image G onto a projection target SC.

[0012] The projection target SC is formed, for example, by the surface of an object such as a screen. In the example shown in Fig. 1, the projection target SC has a rectangular outer shape. Note that the outer shape of the projection target SC is not limited to the example shown in Fig. 1 and can be any shape. Furthermore, the projection target SC is not limited to a flat surface and may be, for example, a concavely or convexly curved surface.

[0013] As shown in FIG. 1, the system 100 includes a projector 10, a camera 20, and a terminal device 30.

[0014] The projector 10 is a display device that projects a projection image G indicated by video data IMG output from the terminal device 30 onto a projection target SC. In the example shown in FIG. 1, the projection image G is projected onto a rectangular area that covers substantially the entire area of ​​the projection target SC. The projector 10 can also project the projection image G onto an area RP that includes the projection target SC. Note that in FIG. 1, the projection image G is displayed as a shaded area. The projection position and shape of the projection image G onto the projection target SC are not limited to the example shown in FIG. 1 and are arbitrary.

[0015] The projector 10 of this embodiment has a function of controlling the operation of the camera 20 and a function of adjusting the shape of the projection image G using the image pickup result of the camera 20.

[0016] The camera 20 is a digital camera having an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0017] The camera 20 captures an image of the area RC. The area RC is an area that includes the projection image G that is projected onto the projection target SC. In the example shown in Fig. 1, the area RC includes the area RP. Note that the camera 20 may be a component of the projector 10.

[0018] The terminal device 30 is a computer that has a function of supplying video data IMG to the projector 10. In the example shown in Fig. 1, the terminal device 30 is a notebook computer. Note that the terminal device 30 is not limited to a notebook computer, and may be, for example, a desktop computer, a smartphone, a tablet terminal, a video playback device, a DVD (Digital Versatile Disk) player, a Blu-ray Disc player, a hard disk recorder, a television tuner device, a CATV (Cable television) set-top box, a video game console, or the like.

[0019] 1-2.Projector Fig. 2 is a block diagram of a projector 10 used in the projection method according to the first embodiment. In addition to the projector 10, Fig. 2 also shows the connection state of a camera 20 and a terminal device 30 to the projector 10. In the example shown in Fig. 2, the terminal device 30 includes a display device 31. The display device 31 is a display device including various display panels such as a liquid crystal display panel and an organic EL display panel.

[0020] 2, the projector 10 has a storage device 11, a processing device 12, a communication device 13, an image processing circuit 14, an optical device 15, and an operation device 16. These are connected to each other so that they can communicate with each other.

[0021] The storage device 11 is a storage device that stores programs executed by the processing device 12 and data processed by the processing device 12. The storage device 11 includes, for example, a hard disk drive or a semiconductor memory. Note that part or all of the storage device 11 may be provided in an external storage device or server outside the projector 10.

[0022] The storage device 11 stores a program PR1, first pattern information DG1, structured light information DG0, first imaging data D1, and correspondence information DC.

[0023] The program PR1 is a program for executing a projection method, which will be described in detail later.

[0024] The first pattern information DG1 is information indicating a first pattern G1, which will be described later. The first pattern G1 is an image of a uniform first color, and is projected onto the projection target SC by the projector 10. The first color is not particularly limited, but is, for example, white or black. The first pattern information DG1 may be included in the structured light information DG0.

[0025] The first imaging data D1 is information indicating a captured image obtained by using the camera 20 to capture an image of the first pattern G1 projected onto the projection target SC.

[0026] The structured light information DG0-1 to DG0-M is information indicating M structured light G0, which is a pattern image used in the structured light method. Here, M is a natural number equal to or greater than 2. The pattern of the structured light G0 is not particularly limited, but examples thereof include a phase shift pattern, a binary code pattern, a dot pattern, a rectangular pattern, a polygonal pattern, a checkered pattern, a Gray code pattern, and a random dot pattern. Hereinafter, the structured light information DG0-1 to DG0-M may be referred to as "structured light information DG0" without distinction. Note that the number of structured light information DG0, i.e., the specific value of M, is not particularly limited, but is, for example, 46 when the pattern of the structured light G0 is a Gray code pattern. The larger the value of M, the more pronounced the effect of the present disclosure.

[0027] The imaging data D0-1 to D0-M are information indicating captured images obtained by capturing, with the camera 20, each of the M structured light beams G0 sequentially projected onto the projection target SC. Hereinafter, the imaging data D0-1 to D0-M may be referred to as "imaging data D0" without distinction.

[0028] The correspondence information DC is information indicating the correspondence relationship between the coordinates of the display coordinate system of the projector 10 and the coordinates of the imaging coordinate system of the camera 20. The display coordinate system of the projector 10 is a coordinate system for expressing the coordinate values ​​of pixels of the display panel 15b described below. The imaging coordinate system of the camera 20 is a coordinate system for expressing the coordinate values ​​of pixels of the imaging element of the camera 20.

[0029] The processing device 12 has the function of controlling each part of the projector 10 and the function of processing various data. The processing device 12 includes, for example, a processor such as a CPU (Central Processing Unit). The processing device 12 may be configured with a single processor or multiple processors. Some or all of the functions of the processing device 12 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 12 may also be integrated with at least a part of the image processing circuit 14.

[0030] The communication device 13 is a communication device capable of communicating with various devices, and acquires video data IMG from the terminal device 30 and communicates with the camera 20. For example, the communication device 13 is a wired communication device such as a wired LAN (Local Area Network), USB (Universal Serial Bus), or HDMI (High Definition Multimedia Interface), or a wireless communication device such as LPWA (Low Power Wide Area), wireless LAN including Wi-Fi, or Bluetooth. "HDMI," "Wi-Fi," and "Bluetooth" are all registered trademarks.

[0031] The image processing circuit 14 is a circuit that performs necessary processing on the video data IMG from the communication device 13 and inputs the data to the optical device 15. The image processing circuit 14 has, for example, a frame memory (not shown), and loads the video data IMG into the frame memory, appropriately performs various processes such as resolution conversion, resizing, and distortion correction, and inputs the data to the optical device 15. The various processes here use the aforementioned correspondence information DC as appropriate. Note that the image processing circuit 14 may also perform processes such as OSD (On Screen Display) processing, where necessary, that generate image information for menu display or operation guides, etc., and combine it with the video data IMG.

[0032] The optical device 15 is a device that projects image light onto the projection target SC. The optical device 15 includes at least a light source 15a, a display panel 15b, and an optical system 15c.

[0033] The light source 15a includes a light source such as a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, and emits red, green, and blue light, respectively. The display panel 15b is an optical modulator including three light modulation elements corresponding to red, green, and blue. Each light modulation element includes, for example, a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (Digital Mirror Device), and generates image light of each color by modulating the corresponding color light. The image light of each color generated by the display panel 15b is combined by a color synthesis optical system to become full-color image light. The optical system 15c is a projection optical system including a projection lens and the like that focuses and projects the full-color image light from the display panel 15b onto the projection target SC. Note that the optical device 15 may include an LED module including one or more LEDs instead of the display panel 15b. In this case, the optical device 15 may be omitted. In this case, an image based on the video data IMG is drawn on the LED module.

[0034] The operation device 16 is a device that accepts operations from the user. For example, the operation device 16 includes an operation panel and a remote control receiver, not shown. The operation panel is provided on the exterior housing of the projector 10, and outputs a signal based on an operation from the user. The remote control receiver receives an infrared signal from a remote control, not shown, decodes the infrared signal, and outputs a signal based on the operation of the remote control. The operation device 16 may be provided as needed, or may be omitted.

[0035] In the above-described projector 10, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a generation unit 12c by executing the program PR1 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the imaging control unit 12b, and the generation unit 12c.

[0036] The projection control unit 12a controls the operations of the image processing circuit 14 and the optical device 15. More specifically, the projection control unit 12a controls the operation of the optical device 15 to project a projection image G onto the projection target SC. More specifically, the projection control unit 12a causes the optical device 15 to project a first pattern G1 (described below) based on first pattern information DG1 onto the projection target SC, and to project a second image G2 (described below) based on structured light information DG0 onto the projection target SC.

[0037] The imaging control unit 12b controls the operation of the camera 20. More specifically, the imaging control unit 12b acquires first imaging data D1 by having the camera 20 capture an image of the first pattern projected onto the projection target SC, and acquires imaging data D0 by having the camera 20 capture an image of the structured light projected onto the projection target SC. The imaging control unit 12b then stores the acquired first imaging data D1 and imaging data D0 in the storage device 11.

[0038] The generating unit 12c generates the correspondence information DC based on the imaging data D0, and adds processing necessary for generating the correspondence information DC based on the first imaging data D1 as appropriate.

[0039] 1-3. Projection method 3 is a flowchart showing the flow of the projection method according to the first embodiment. The projection method is performed by using the system 100 described above, in which the processing device 12, which is an example of a "computer," executes the program PR1.

[0040] The projection method of this embodiment includes steps S10 to S90, as shown in Fig. 3. Here, as described above, the projector 10 includes the optical device 15 and the processing device 12 that controls the operation of the optical device 15, and the processing device 12 executes steps S10 to S90. Also, the program PR1 causes the processing device 12 to execute steps S10 to S90.

[0041] The processing device 12 first executes step S10. Step S10 executes projecting a user interface image GU from the projector 10. The user interface image GU includes an item for setting a first number of times, which is the determination criterion for step S60. A specific example of the user interface image GU will be described later with reference to FIG. 4.

[0042] After step S10, the processing device 12 executes step S20. Step S20 measures the projection range onto which each of the M beams of structured light GS is projected on the projection target SC. Specifically, step S20 includes steps S21, S22, and S23, in this order.

[0043] Prior to step S30, i.e., before the projector 10 sequentially projects M beams of structured light GS onto the projection target SC, the processing device 12 executes step S21 in which the projector 10 projects a first pattern G1 (described below) onto the projection target SC. This projection is performed by the projection control unit 12a controlling the operations of the image processing circuit 14 and the optical device 15 based on first pattern information DG1. The first pattern G1 is a uniform image of a first color. Details of the first pattern G1 will be described later with reference to FIG. 5.

[0044] In step S22, the processing device 12 executes the process of capturing an image of a first pattern G1 (described later) projected from the projector 10 onto the projection target SC by the camera 20. This capturing is performed by the imaging control unit 12b controlling the operation of the camera 20. As a result of this capturing, first captured data D1 indicating the capturing result is generated, and the first captured data D1 is stored in the storage device 11.

[0045] In step S23, the processing device 12 determines a projection range RP1 (described later) onto which each of the M pieces of structured light GS on the projection target SC is projected, based on the first pattern G1 indicated by the first imaging data D1.

[0046] After step S20, processing device 12 executes step S30. Step S30 involves acquiring a plurality of pieces of imaging data D0 by using camera 20 to capture M (M is a natural number equal to or greater than 2) pieces of structured light GS projected from projector 10 onto projection target SC. Specifically, step S30 includes steps S31 and S32.

[0047] In step S31, processing device 12 executes sequential projection of M beams of structured light GS (described below) onto projection target SC from projector 10. This projection is performed by projection control unit 12a controlling the operations of image processing circuit 14 and optical device 15 based on structured light information DG0-1 to DG0-M.

[0048] In step S32, the processing device 12 acquires M pieces of imaging data D0, i.e., imaging data D0-1 to D0-M, by capturing, with the camera 20, M pieces of structured light GS (described below) projected from the projector 10 onto the projection target SC. This acquisition is performed by the imaging control unit 12b controlling the operation of the camera 20. As a result of this acquisition, M pieces of imaging data D0 are generated, and the generated M pieces of imaging data D0 are stored in the storage device 11.

[0049] In step S30 described above, after executing steps S31 and S32, the processing device 12 associates the coordinates of the display coordinate system of the projector 10 with the coordinates of the imaging coordinate system of the camera 20 based on the imaging data D0-1 to D0-M. This association is performed by the generation unit 12c based on the imaging data D0-1 to D0-M. For example, the processing device 12 performs the following operations to associate the coordinates: searching for a feature point of the structured light GS included in the imaging data D0-1, acquiring the coordinate value of the feature point in the imaging coordinate system, and identifying the coordinate value of the pixel on the display panel 15b that corresponds to the coordinate value of the feature point in the imaging coordinate system. The processing device 12 also performs a similar calculation on the imaging data D0-2 to D0-M. Correspondence information DC is generated by this association, and the generated correspondence information DC is stored in the storage device 11. That is, in this embodiment, steps S31 and S32 are alternately repeated in step S30. Therefore, in step S30, the projection of structured light GS and the capture of structured light GS are performed M times each. Note that this association need only be performed by the end of the processing shown in Fig. 3, and does not have to be performed in step S30. For example, this association may be performed when it is determined in step S50 (described later) that there is no first structured light GS1, or may be performed in step S90 (described later).

[0050] After step S30, processing device 12 executes step S40. In step S40, processing is executed to detect a first structured light GS1, which is a structured light GS that has not been properly captured among M pieces of structured light GS, based on the captured image data D0-1 to D0-M. This detection will be described later with reference to FIGS. 6 to 8.

[0051] After step S40, the processing device 12 executes step S50. In step S50, the processing device 12 determines, using the processing result of step S40 and based on the plurality of pieces of imaging data D0, whether or not there is at least one first structured light GS1 among the M pieces of structured light GS that has not been properly captured.

[0052] In this embodiment, the determination in step S50 of whether at least one first structured light GS1 exists is performed based on each of the M structured lights GS in a projection range RP1 (described later). Therefore, even if there is a structured light GS among the M structured lights GS that is not properly captured outside the projection range RP1 (described later), the processing device 12 determines that the structured light GS does not correspond to the first structured light GS1.

[0053] When the processing device 12 determines that at least one first structured light GS1 exists (step S50: YES), it executes step S60. Step S60 determines whether N pieces of structured light GS have been repeatedly projected a first number of times or more in step S90. That is, step S60 determines whether the number of times that N pieces of structured light GS have been repeatedly projected in step S90 is a first number of times or more. The first number of times is not particularly limited and is arbitrary. In this embodiment, the first number of times can be changed using a user interface image GU, which will be described later. Note that step S60 may be omitted.

[0054] If the number of times that the projection of N sheets of structured light GS in step S90 is repeated is equal to or greater than the first number of times (step S60: YES), the processing device 12 executes step S70. Step S70 executes projecting messages Rd and Re, which will be described later, from the projector 10. Specific examples of the messages Rd and Re will be described later with reference to FIG. 9. Note that step 70 may be omitted.

[0055] After step S70, the processing device 12 executes step S80. In step S80, it is determined whether or not to execute step S90 based on the input result for the image GM, which will be described later. Note that step S80 may be omitted.

[0056] If the number of times that the projection of N structured light beams GS in step S90 is repeated is less than the first number of times (step S60: NO), or if it is determined that step S90 should be repeated (step S80: YES), the processing device 12 executes step S90. In step S90, the processing device 12 causes the optical device 15 to project N (N is a natural number greater than 0 and less than M) structured light beams GS including at least one first structured light beam GS1 onto the projection target SC. Specifically, step S90 includes steps S91 and S92.

[0057] Step S91 executes sequential projection of N pieces of structured light GS, including first structured light GS1, which is the structured light GS that could not be properly captured among the M pieces of structured light GS, onto the projection target SC from the projector 10. This projection is performed by the projection control unit 12a controlling the operation of the image processing circuit 14 and the optical device 15 based on the structured light information DG0-1 to DG0-M.

[0058] In step S92, N pieces of imaging data D0 are acquired by capturing images of the N pieces of structured light GS projected from the projector 10 onto the projection target SC with the camera 20. This acquisition is performed by the imaging control unit 12b controlling the operation of the camera 20. Through this acquisition, N pieces of imaging data D0 are generated, and the generated N pieces of imaging data D0 are stored in the storage device 11. At this time, of the M pieces of imaging data D0 already stored in the storage device 11, the imaging data D0 corresponding to the N pieces of imaging data D0 generated in step S92 is updated by overwriting or the like.

[0059] After step S90, the processing device 12 returns to step S40, whereby steps S40 and S50 are executed again.

[0060] If the processing device 12 determines that the first structured light GS1, which is the structured light GS that was not properly captured, does not exist (step S50: NO), or if it determines not to repeat step S90 (step S80: NO), it terminates the process. Note that if the processing device 12 determines that the first structured light GS1 does not exist after executing step S90 (step S50: NO), it performs the association using the captured image of the second structured light GS2, which is the structured light GS that was properly captured in step S30 among the M structured light GSs, and the captured image of the third structured light GS3, which is the structured light GS that was properly captured in step S90. That is, if the processing device 12 determines that the first structured light GS1 does not exist after executing step S90 (step S50: NO), when performing the association, it reuses the captured image of the second structured light GS2, which was properly captured in step S30 among the M structured light GSs after step S50.

[0061] 4 is a diagram for explaining the user interface image GU. In step S10, for example, as shown in FIG.

[0062] The user interface image GU is a projection image G for receiving operations from the user. In the example shown in Fig. 4, the user interface image GU includes areas Ra and Rb.

[0063] Area Ra has buttons B1 and B2. Button B1 is a display for starting the processing from step S20 onwards. Button B2 is a display for setting the value of the first number of times in step S60. Note that button B2 for changing the first number of times is not limited to being included in user interface image GU, and may be included in image GM, which will be described later, for example.

[0064] The area Rb is a display that explains how to operate the operation device 16 with respect to the user interface image GU.

[0065] In the above user interface image GU, when button B1 is operated, the display of user interface image GU ends, and then execution of step S20 begins. Also, when button B2 is operated, a widget for changing the value of the first number of times is displayed. When the value of the first number of times is changed, the display of user interface image GU ends, and then execution of step S20 begins.

[0066] Fig. 5 is a diagram for explaining the first pattern G-1. In step S21, as shown in Fig. 5, the first pattern G-1 indicated by the first pattern information DG1 is projected from the projector 10 onto the projection target SC. Note that in Fig. 5, the outer edge of the first pattern G-1 is located inside the outer edge of the region RP, but the outer edge of the first pattern G-1 may also coincide with the outer edge of the region RP.

[0067] In step S22, the processing device 12 causes the camera 20 to capture an image of the first pattern G-1 in the region RC that includes the first pattern G-1 projected onto the projection target SC.

[0068] In step S23, the processing device 12 determines the projection range RP1 by detecting the edges of the first pattern G-1 in the image shown by the imaging result of the first pattern G-1. Therefore, the processing device 12 determines the range into which the first pattern G-1 is projected as the projection range RP1. In this embodiment, the first pattern G-1 is a pattern that uses all the pixels of the display panel 15b. The first pattern G-1 may also be a pattern that uses some of the pixels of the display panel 15b. For example, the processing device 12 may change the size of the first pattern G-1 as appropriate based on size information such as the aspect ratio of the video data IMG.

[0069] Fig. 6 is a diagram for explaining anomaly detection when a Gray code pattern is used as the structured light GS. Fig. 6 shows, as a representative example, two structured light GSs, structured light GS-a and GS-b, which are inverted images of each other, among M structured light GSs when a Gray code pattern is used as the structured light GS. Structured light GS-b is a pattern in which the bright and dark areas of structured light GS-a are inverted.

[0070] In the Gray code method, as shown in Fig. 6, robustness may be enhanced by using the difference between an image captured with structured light GS-a, which is a normal pattern, and an image captured with structured light GS-b, which is an inverted pattern. For example, if the difference is greater than zero, the image is determined to be white, and if the difference is less than zero, the image is determined to be black. The Gray code method uses multiple pairs of such normal patterns and inverted patterns.

[0071] The difference between the captured image of such a normal pattern and the captured image of the inverted pattern changes when a person or their shadow temporarily exists between the projector 10 and the projection target SC, or when the amount of light, such as lighting in the space where the projector 10 is installed or light from a window, changes.

[0072] Therefore, when a Gray code pattern is used as the structured light GS, in step S40, for each pair of a normal pattern and an inverted pattern, the absolute value of the difference between the pixel value of a pixel constituting the captured image of the normal pattern and the pixel value of a pixel constituting the captured image of the inverted pattern is calculated for each pixel, and the number of pixels n1, which is the number of pixels whose absolute value is equal to or greater than a threshold value t1, and the number of pixels n2, which is the number of pixels whose absolute value is less than the threshold value t1, are counted. The pixel value is a value that characterizes the color or brightness of a pixel, such as a gradation value or luminance value. The number of pixels n1, which has a large absolute value, is the number of pixels that are relatively reliable. The number of pixels n2, which has a small absolute value, is the number of pixels that are relatively unreliable. Since the normal pattern and the inverted pattern are simply inverted colors, all pixels should ideally satisfy the condition of being equal to or greater than the threshold value t1. However, when the normal pattern and the inverted pattern are captured, for example, a person's shadow may overlap a bright area of ​​the structured light GS-a, reducing the brightness of the bright area to a brightness similar to that of the dark area. In such a case, even if a set of a normal pattern and an inverted pattern is used, pixels that are less than the threshold value t1 will occur, and pixels that should be judged as white will be judged as black.

[0073] In step S50, the processing device 12 determines whether the ratio r=n2 / n1 of the number of pixels n2 to the number of pixels n1 before and after the pair of normal and inverted patterns is equal to or greater than threshold value t2. Specifically, when the pair of normal and inverted patterns projected during a first period is defined as a first pair, and the pair of normal and inverted patterns projected during a second period after the first period is defined as a second pair, if the ratio q of the ratio r2 in the second pair to the ratio r1 in the first pair is equal to or greater than threshold value t2, it is determined that there is structured light GS among the M pieces of structured light GS that has not been captured correctly. For example, if the number of pixels n1=100 and the number of pixels n2=50 during the first period, then r1=0.5. If the number of pixels n1=80 and the number of pixels n2=70 during the second period, then r2=0.825. In this case, for example, if the threshold value t2 is 0.51 / 0.5=1.02, then q = r2 / r1 = 1.65 ≥ t2 is satisfied, and the processing device 12 can determine that a non-negligible abnormality has occurred in the second period. That is, in this embodiment, in step S50, the processing device 12 detects whether the ratio q of the ratio r (the ratio of the number of pixels n2 to the number of pixels n1) between adjacent pairs of multiple normal and inverted patterns has changed by a predetermined amount. Normally, the ratio r should hardly change between adjacent pairs. Therefore, the processing device 12 in this embodiment determines whether the number of pixels n2 with hypothetical low reliability has increased based on whether the ratio r has changed by a predetermined amount. Here, the structured light GS for a pair of normal and inverted patterns for which the ratio q is equal to or greater than the threshold value t2 is the first structured light GS1. On the other hand, if the ratio q is less than the threshold value t2 for all pairs of normal and inverted patterns, it is determined that there is no structured light GS that has not been properly captured among the M structured light GS. The processing device 12 may determine whether or not there is structured light GS that has not been captured normally, based on a change in the absolute number of pixels n2.

[0074] Fig. 7 is a diagram for explaining the detection of anomalies when a phase-shift pattern is used as the structured light GS. Fig. 7 shows, as a representative example, two structured lights GS, structured lights GS-c and GS-d, which are inverted images of each other, among M structured lights GS when a phase-shift pattern is used as the structured light GS. Fig. 8 is a diagram for explaining the detection of anomalies when a phase-shift pattern is used as the structured light GS.

[0075] The phase shift method uses multiple phase shift patterns that have a stripe pattern in which brightness values ​​change along a sine wave and are out of phase with each other, as shown in Fig. 7. Therefore, when structured light GS-c and structured light GS-d are captured, if normal, the pixel values ​​of the captured images also change along a sine wave.

[0076] Therefore, when a phase shift pattern is used as the structured light GS, in step S40, the pixel values ​​(measurement values) of the captured image of each structured light GS are fitted with a sine wave, and the difference between the fitting result and the pixel value of the captured image is calculated for each pixel, and then the total value of these differences is calculated. In the example shown in Fig. 8, one measurement value deviates significantly from the fitting result. Note that in Fig. 8, the measurement values ​​are indicated by dots, and the fitting result is indicated by a solid line.

[0077] In step S50, the processing device 12 determines whether the change in the total difference value before and after the projection order is equal to or greater than threshold t3. If the change is equal to or greater than threshold t3, the processing device 12 determines that there is structured light GS among the M pieces of structured light GS that has not been properly captured. On the other hand, if the change is less than threshold t3, the processing device 12 determines that there is no structured light GS among the M pieces of structured light GS that has not been properly captured.

[0078] The method for determining whether or not there is structured light GS that has not been properly captured is not limited to the above example. For example, the method may be based on the detection results of a motion sensor that detects the passage of a person between the projector 10 and the projection target SC, or on the detection results of an illuminance sensor that measures the illuminance in the installation space of the projection target SC. In an embodiment in which such a sensor is used to detect the presence of structured light GS that has not been properly captured, the detection is performed over the period in which M pieces of structured light GS are projected by the projector 10. If the presence of structured light GS that has not been properly captured is detected, N pieces of structured light GS, including the structured light GS at the time of detection, may be projected again. Alternatively, the projection of structured light GS may be suspended over the period in which the sensors detect the passage of a person or a change in illuminance, and the projection of structured light GS may be resumed after the detection ceases.

[0079] Fig. 9 is a diagram showing an example of the messages Rd and Re. Fig. 9 shows the image GM that the projector 10 projects onto the projection target SC in step S70.

[0080] The image GM is the projection image G that the projector 10 projects onto the projection target SC in step S70. The image GM includes an area Rc and messages Rd and Re.

[0081] Area Rc has buttons B3 and B4. Button B3 is a display for permitting execution of step S90. Button B4 is a display for not permitting execution of step S90.

[0082] The message Rd indicates that even if N pieces of structured light GS are projected, there is a possibility that at least one first structured light GS1 will be generated. In the example shown in Fig. 9, the message Rd displays the words "Even if you continue, an abnormality may occur." Note that the content of the message Rd is not limited to the example shown in Fig. 9 and may be arbitrary.

[0083] The message Re includes one or more possible causes related to the inclusion of at least one first structured light GS1. In the example shown in Fig. 9, the message Re displays the text "Please check for obstacles before proceeding." Note that the displayed content and number of messages Re are not limited to the example shown in Fig. 9 and are arbitrary.

[0084] In the above image GM, when button B3 is operated, step S80 is executed, and it is determined in step S80 that step S90 should be repeated. Also, when button B4 is operated, step S80 is not executed and the process ends.

[0085] As described above, the above projection method includes steps S30, S50, and S90. If, as a result of capturing M pieces of structured light GS in step S30, there is at least one first structured light GS1 that has not been captured properly (step S50: YES), the above projection method includes projecting N pieces of structured light GS in step S90, which N pieces of structured light GS include the at least one first structured light GS1 and is smaller than M. This makes it possible to minimize the number of pieces of structured light GS that are re-projected. As a result, even if there is a structured light GS that has not been captured properly due to disturbance or the like, the number of pieces of structured light GS that are projected can be optimized.

[0086] For example, if 46 patterns using the Gray code method are used as structured light GS, and three patterns have abnormalities and it takes one second to capture one image, then conventionally, the first structured light measurement would take 46 seconds, and another 46 seconds to redo the measurement, for a total of 92 seconds. In contrast, with the projection method of the present disclosure, the redo of the measurement can be completed in as little as three seconds, for a total of 49 seconds.

[0087] As described above, the projection method of this embodiment includes steps S60 and S70. In step S70, if it is determined that N pieces of structured light GS have been repeatedly projected the first number of times or more and that at least one first structured light GS1 exists (step S50: YES, step S60: YES), the optical device 15 is caused to project a message Rd indicating that there is a possibility that at least one first structured light GS1 will be generated even after projecting N pieces of structured light GS. This allows the user to determine through the message Rd whether or not to further project N pieces of structured light GS the first number of times.

[0088] Furthermore, as described above, in step S70, if it is determined that N pieces of structured light GS have been repeatedly projected a first number of times or more and that at least one first structured light GS1 is present (step S50: YES, step S60: YES), a message Re including one or more possible causes for the inclusion of at least one first structured light GS1 is projected from the projector 10. This allows the user to easily understand the one or more possible causes through the message Re. As a result, the user can be prompted to make adjustments to prevent the occurrence of an incorrectly captured first structured light GS1.

[0089] Furthermore, as described above, the projection method of this embodiment includes step S10. Step S10 involves projecting a user interface image GU for setting the first number of times from the projector 10. This allows the number of attempts to project N beams of structured light GS to be adjusted according to the user's wishes.

[0090] As described above, the projection method of this embodiment includes steps S21 and S23. In step S21, before sequentially projecting M pieces of structured light GS onto the projection target SC from the projector 10, a first pattern G1 of a uniform first color is projected from the projector 10. In step S23, a projection range on the projection target SC onto which each of the M pieces of structured light GS is projected is determined based on the first pattern G1. In step S50, determining whether at least one first structured light GS1 is present is performed based on each of the M pieces of structured light GS within the projection range. This makes it possible to determine whether or not a first structured light GS1 that is not properly captured is present without being affected by objects such as foreign matter outside the projection range.

[0091] 2. Second embodiment A second embodiment of the present disclosure will be described below. In the following exemplary embodiment, for elements whose actions and functions are similar to those of the first embodiment, the reference numerals used in the description of the first embodiment will be used, and detailed descriptions of each element will be omitted as appropriate.

[0092] 10 is a block diagram of a projector 10A used in a projection method according to the second embodiment. The projector 10A of this embodiment is configured similarly to the projector 10 of the first embodiment, except that a program PR2 is used instead of the program PR1 of the first embodiment. Note that a system 100A used in a projection method according to the second embodiment is configured similarly to the system 100 of the first embodiment, except that a projector 10A is used instead of the projector 10.

[0093] In the projector 10A, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a generation unit 12d by executing the program PR2 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the imaging control unit 12b, and the generation unit 12d.

[0094] The generation unit 12d is similar to the generation unit 12d of the first embodiment, except that it has the added function of generating change information D3 based on the first imaging data D1 and the second imaging data D2, and the function of performing processing based on the change information D3 when generating the correspondence information DC.

[0095] The second imaging data D2 is information indicating a captured image obtained by capturing an image of the first pattern G1 projected onto the projection target SC with the camera 20, and is acquired at a timing different from that of the first imaging data D1.

[0096] The change information D3 is information that indicates a change in one or both of the position and attitude of the projector 10A relative to the projection target SC. The change information D3 in this embodiment is identified based on the first imaging data D1 and the second imaging data D2.

[0097] 11 is a flowchart showing the flow of the projection method according to the second embodiment. The projection method according to the second embodiment is the same as the projection method according to the first embodiment, except that steps S100, S110, and S120 are added.

[0098] In the projection method of this embodiment, the processing device 12 executes step S100 after step S30. In step S100, similar to step S21, the projector 10A projects the first pattern G1 onto the projection target SC.

[0099] After step S100, the processing device 12 executes step S110. In step S110, similar to step S22, the processing device 12 executes the process of capturing an image of the first pattern G1 projected onto the projection target SC from the projector 10A in step S110 with the camera 20. By capturing this image, second captured image data D2 indicating the captured image data is generated, and the second captured image data D2 is stored in the storage device 11.

[0100] After step S110, the processing device 12 executes step S40. Then, if the processing device 12 determines that at least one first structured light GS1 exists (step S50: YES), the processing device 12 executes step S120.

[0101] In step S120, the processing device 12 determines whether or not one or both of the position and attitude of the projector 10A relative to the projection target SC has changed, based on the change information D3.

[0102] In this embodiment, the change information D3 is identified based on the first imaging data D1 and the second imaging data D2. Specifically, in step S120, for example, the difference between the pixel values ​​of the image indicated by the first imaging data D1 and the pixel values ​​of the image indicated by the second imaging data D2 is calculated, and the calculation result is used as the change information D3. In step S120, if the difference is equal to or greater than a predetermined threshold, it is determined that one or both of the position and orientation of the projector 10A relative to the projection target SC has changed. On the other hand, if the difference is less than the predetermined threshold, it is determined that one or both of the position and orientation of the projector 10A relative to the projection target SC has not changed.

[0103] If it is determined that one or both of the position and attitude of the projector 10A relative to the projection target SC has changed (step S120: YES), the processing device 12 returns to step S20, whereby the structured light method using M structured light beams GS in step S30 is executed again.

[0104] On the other hand, if it is determined that one or both of the position and attitude of the projector 10A relative to the projection target SC has not changed (step S120: NO), the processing device 12 proceeds to step S60. As a result, the structured light method using M structured light beams GS in step S30 is not executed, and the structured light method using N structured light beams GS in step S90 is executed.

[0105] The above projection method also makes it possible to optimize the number of projected structured light beams GS, even when there are structured light beams GS that are not properly captured due to disturbances or the like. In this embodiment, as described above, if, as a result of capturing M structured light beams GS, it is determined that at least one first structured light beam GS1 is not properly captured and one or both of the position and the orientation of the projector 10A relative to the projection target SC have not changed (step S50: YES, step S120: NO), N structured light beams GS that include the at least one first structured light beam GS1 and that are smaller than M are projected, thereby minimizing the number of structured light beams GS to be re-projected. On the other hand, if, as a result of capturing M structured light beams GS, it is determined that at least one first structured light beam GS1 is not properly captured and one or both of the position and the orientation have changed (step S50: YES, step S120: YES), all M structured light beams GS are projected, thereby minimizing the number of structured light beams GS to be re-projected. From the above, even if there is structured light GS that is not captured normally due to disturbance or the like, the number of projected structured light GS can be optimized.

[0106] If one or both of the position and attitude of projector 10 relative to projection target SC changes, the captured images of structured light GS captured correctly in step S30 may not be usable for the correspondence performed by processing device 12 after step S50, for example. This is because if one or both of the position and attitude change, the position at which structured light GS is projected on projection target SC changes, and the conditions for capturing structured light GS by camera 20 change before and after one or both of the position and attitude change. Therefore, in order to improve the accuracy of the calculation of correspondence by processing device 12, it is preferable to project all M images of structured light GS when one or both of the position and attitude change.

[0107] As described above, in step S22, before the projector 10A sequentially projects M beams of structured light GS onto the projection target SC, the camera 20 captures an image of the first pattern G1 of a uniform first color projected from the projector 10A onto the projection target SC, thereby acquiring first imaging data D1. In step S110, after the projector 10A sequentially projects M beams of structured light GS onto the projection target SC and before determining whether at least one beam of first structured light GS1 is present, the camera 20 captures an image of the first pattern G1 projected from the projector 10A onto the projection target SC, thereby acquiring second imaging data D2. Then, the change information D3 is identified based on the first imaging data D1 and the second imaging data D2. This makes it possible to determine whether or not there has been a change in one or both of the position and attitude of the projector 10A relative to the projection target SC, without using a sensor other than the camera 20.

[0108] 3. Third embodiment A third embodiment of the present disclosure will be described below. In the following exemplary embodiments, elements whose actions and functions are similar to those of the first embodiment will be designated by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions of each element will be omitted where appropriate.

[0109] 12 is a block diagram of a projector 10B used in a projection method according to the third embodiment. The projector 10B of this embodiment uses a program PR3 instead of the program PR1 of the first embodiment, and is configured similarly to the projector 10 of the first embodiment, except that a sensor 17 is added. Note that a system 100B used in a projection method according to the third embodiment is configured similarly to the system 100 of the first embodiment, except that a projector 10B is used instead of the projector 10.

[0110] The sensor 17 is a sensor that detects changes in one or both of the position and attitude of the projector 10B relative to the projection target SC, and is installed on one or both of the projection target SC and the projector 10B. For example, an angular velocity sensor, an acceleration sensor, etc. are used as the sensor 17. The detection result of the sensor 17 is stored in the storage device 11 as first information DS1 or second information DS2.

[0111] In the projector 10B, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a generation unit 12e by executing the program PR3 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the imaging control unit 12b, and the generation unit 12e.

[0112] The generation unit 12e is similar to the generation unit 12d of the first embodiment, except that it has the added function of generating change information D3 based on the first information DS1 and the second information DS2, and the function of performing processing based on the change information D3 when generating the correspondence information DC.

[0113] The first information DS1 is information that indicates one or both of the attitude and position of the projector 10B with respect to the projection target SC before the projector 10B sequentially projects M pieces of structured light GS onto the projection target SC. The second information DS2 is information that indicates one or both of the attitude and position of the projector 10B relative to the projection target SC after the projector 10B sequentially projects M pieces of structured light GS onto the projection target SC and before determining whether at least one first structured light GS1 is present.

[0114] The change information D3 of this embodiment is identified based on the first information DS1 and the second information DS2.

[0115] 13 is a flowchart showing the flow of the projection method according to the third embodiment. The projection method according to the third embodiment is the same as the projection method according to the first embodiment, except that steps S120B, S130, and S140 are added.

[0116] In the projection method of this embodiment, the processing device 12 executes step S130 after step S20. Step S130 executes acquiring first information DS1 based on the detection result of the sensor 17. The acquired first information DS1 is stored in the storage device 11.

[0117] After step S130, the processing device 12 executes step S30. Then, after step S30, the processing device 12 executes step S140. Step S140 executes acquiring second information DS2 based on the detection result of the sensor 17. The acquired second information DS2 is stored in the storage device 11.

[0118] After step S140, the processing device 12 executes step S40. Then, if the processing device 12 determines that at least one first structured light GS1 exists (step S50: YES), the processing device 12 executes step S120B.

[0119] Step S120B executes a determination as to whether or not one or both of the position and attitude of the projector 10B relative to the projection target SC has changed, based on the change information D3.

[0120] In this embodiment, change information D3 is identified based on the first information DS1 and the second information DS2. Specifically, in step S120B, for example, the difference between the detection value indicated by the first information DS1 and the detection value indicated by the second information DS2 is calculated, and the calculation result is used as change information D3. In step S120B, if the difference is equal to or greater than a predetermined threshold, it is determined that one or both of the position and orientation of projector 10B relative to the projection target SC have changed. On the other hand, if the difference is less than the predetermined threshold, it is determined that one or both of the position and orientation of projector 10B relative to the projection target SC have not changed.

[0121] If it is determined that one or both of the position and attitude of the projector 10B relative to the projection target SC has changed (step S120B: YES), the processing device 12 returns to step S20, whereby the structured light method using M structured light beams GS in step S30 is executed again.

[0122] On the other hand, if it is determined that one or both of the position and attitude of the projector 10B relative to the projection target SC has not changed (step S120B: NO), the processing device 12 proceeds to step S60. As a result, the structured light method using M structured light beams GS in step S30 is not executed, and the structured light method using N structured light beams GS in step S90 is executed.

[0123] The above projection method also makes it possible to optimize the number of projections of structured light GS, even when there is structured light GS that is not properly captured due to disturbances or the like. In this embodiment, in step S130, first information DS1 is acquired, indicating one or both of the attitude and position of the projector 10B before sequentially projecting M pieces of structured light GS onto the projection target SC. In addition, in step S140, second information DS2 is acquired, indicating one or both of the attitude and position of the projector 10B after sequentially projecting M pieces of structured light GS onto the projection target SC and before determining whether at least one first structured light GS1 is present. Then, change information D3 is identified based on the first information DS1 and the second information DS2. This makes it possible to determine whether there has been a change in one or both of the position and attitude of the projector 10B relative to the projection target SC, while reducing the number of projections from the projector 10B.

[0124] 4. Variations The above-described embodiments can be modified in various ways. Specific modifications that can be applied to the above-described embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples can be combined as appropriate within the scope of not mutually contradictory.

[0125] 4-1. Variation 1 In the above-described embodiment, an example is given in which the processing device 12 of the projectors 10, 10A, and 10B executes the programs PR1, PR2, and PR3, but this is not limited to this example. For example, the processing device of a computer communicatively connected to the projectors 10, 10A, and 10B and the camera 20 may execute the programs PR1, PR2, and PR3, or the processing device of the camera 20 may execute the programs PR1, PR2, and PR3.

[0126] 4-2. Variation 2 In the above-described embodiment, an example is given in which the correspondence information DC is used to adjust the projection image G, but this is not limited to this example. For example, the correspondence information DC may be used to display a uniform grid-like pattern or the like on the projection target SC, or may be used to reflect a three-dimensional model of the projection target SC as seen from the camera 20 in three-dimensional image editing software or the like, and then draw a picture on the model, and display how the picture looks when viewed from the projector 10 on a PC monitor or the like, or project it on the projector 10.

[0127] 4-3. Variation 3 In the second embodiment described above, the change information D3 is determined based on the first captured image data D1 and the second captured image data D2, and in the third embodiment described above, the change information D3 is determined based on the first information DS1 and the second information DS2. However, the present invention is not limited to this example. For example, the change information D3 may be determined based on a value input by the user. Specifically, when the user inputs a value indicating the change information D3 into the operation device 16, the change information D3 may be determined based on the value or the index. The value may be, for example, the distance between each of the projectors 10, 10A, and 10B and the projection target SC or the installation angle of the projectors 10, 10A, and 10B. Note that the change information D3 does not have to be numerical information. For example, the change information D3 may be event information indicating the occurrence of a change event in at least one of the position and the attitude. The content of the event information is determined based on the result of selecting either "Yes" or "No" when, for example, the optical device 15 projects a confirmation screen containing a message such as "Has the projector body moved?" and two options, "Yes" and "No."

[0128] 5. Supplementary Notes A summary of this disclosure is provided below.

[0129] (Supplementary Note 1) A first aspect, which is a preferred example of the projection method of the present disclosure, includes acquiring a plurality of pieces of imaging data by capturing images of M (M is a natural number greater than or equal to 2) pieces of structured light projected from a projector onto a projection target using a camera, determining based on the plurality of pieces of imaging data whether or not there is at least one first structured light among the M pieces of structured light that has not been properly captured, and, if it is determined that there is at least one first structured light, projecting N (N is a natural number smaller than M) pieces of structured light including the at least one first structured light onto the projection target.

[0130] In the above aspect, if, as a result of capturing M pieces of structured light, there is at least one first structured light that has not been captured properly, N pieces of structured light that include the at least one first structured light and are smaller than M are projected, thereby making it possible to minimize the number of structured light pieces that are re-projected. As a result, even if there is structured light that has not been captured properly due to disturbances or the like, it is possible to optimize the number of projected structured light pieces.

[0131] (Supplementary Note 2) In the second aspect, which is a preferred example of the first aspect, if it is determined that the number of repetitions of projecting the N structured lights is equal to or greater than a first number and that the at least one first structured light is present, the projector may project a message indicating that the at least one first structured light may still be generated even if the N structured lights are projected. In the above aspect, the message may prompt the user to decide whether or not to project the N structured lights a first number of times.

[0132] (Supplementary Note 3) In a third aspect, which is a preferred example of the first or second aspect, the projector further includes projecting a message from the projector containing one or more possible causes for the presence of the at least one first structured light when the number of repetitions of the projection of the N structured light is equal to or greater than a first number and it is determined that the at least one first structured light is present. In the above aspect, the message allows the user to easily understand the one or more possible causes. As a result, the user can be prompted to make adjustments to prevent the occurrence of an improperly captured first structured light.

[0133] (Supplementary Note 4) In a fourth aspect, which is a preferred example of the second or third aspect, the method further includes projecting from the projector a user interface image for setting the first number of times. In the above aspects, the number of attempts to project N structured light beams can be adjusted according to the user's wishes.

[0134] (Supplementary Note 5) In a fifth aspect, which is a preferred example of any of the first to fourth aspects, before sequentially projecting the M pieces of structured light from the projector onto the projection target, the method further includes projecting a first pattern of a uniform first color from the projector onto the projection target, and determining a projection range on the projection target onto which each of the M pieces of structured light is projected based on the first pattern, wherein determining whether or not at least one first structured light is present is performed based on each of the M pieces of structured light in the projection range. In the above aspect, it is possible to determine whether or not there is first structured light that is not properly captured, without being affected by objects such as foreign matter outside the projection range.

[0135] (Supplementary Note 6) A sixth aspect, which is another preferred example of the projection method of the present disclosure, includes: acquiring a plurality of pieces of imaging data by capturing images of M (M is a natural number greater than or equal to 2) pieces of structured light projected from a projector onto a projection target, with a camera; determining, based on the plurality of imaging data, whether or not at least one first structured light among the M pieces of structured light is not properly captured; determining, based on change information indicating a change in one or both of a position and an attitude of the projector relative to the projection target, whether or not the position and / or the attitude have changed; projecting N (N is a natural number smaller than M) pieces of structured light including the at least one first structured light onto the projection target when it is determined that the at least one first structured light is present and one or both of the position and the attitude have not changed; and projecting the M pieces of structured light onto the projection target when it is determined that the at least one first structured light is present and one or both of the position and the attitude have changed.

[0136] In the above aspect, if, as a result of capturing M pieces of structured light, it is determined that at least one first structured light is not captured correctly and one or both of the position and the attitude have not changed, N pieces of structured light including the at least one first structured light and smaller than M are projected, thereby minimizing the number of structured light beams to be re-projected. On the other hand, if, as a result of capturing M pieces of structured light, it is determined that at least one first structured light is not captured correctly and one or both of the position and the attitude have changed, all M pieces of structured light are projected, thereby minimizing the number of structured light beams to be re-projected. From the above, it is possible to optimize the number of projected structured light beams even when there is structured light that is not captured correctly due to disturbances or the like.

[0137] (Supplementary Note 7) In a seventh aspect, which is a preferred example of the sixth aspect, the method includes: acquiring first imaging data by capturing an image of a first pattern of a uniform first color projected from the projector onto the projection target with the camera before sequentially projecting the M pieces of structured light from the projector onto the projection target; and acquiring second imaging data by capturing an image of the first pattern projected from the projector onto the projection target with the camera after sequentially projecting the M pieces of structured light from the projector onto the projection target and before determining whether or not the at least one first structured light is present, wherein the change information is identified based on the first imaging data and the second imaging data. In the above aspect, it is possible to determine whether or not there has been a change in one or both of the position and attitude of the projector relative to the projection target without using a sensor separate from the camera.

[0138] (Supplementary Note 8) In the eighth aspect, which is a preferred example of the sixth aspect, the method includes acquiring first information indicating one or both of the attitude and the position before the M pieces of structured light are sequentially projected from the projector onto the projection target, and acquiring second information indicating one or both of the attitude and the position after the M pieces of structured light are sequentially projected from the projector onto the projection target and before determining whether the at least one first structured light is present, wherein the change information is identified based on the first information and the second information. In the above aspect, it is possible to determine whether one or both of the position and attitude of the projector relative to the projection target has changed while reducing the number of projections from the projector.

[0139] (Appendix 9) A ninth aspect, which is a preferred example of the projector of the present disclosure, includes an optical device and a processing device that controls the operation of the optical device, and the processing device performs the following operations: acquires a plurality of pieces of imaging data in which M (M is a natural number greater than or equal to 2) pieces of structured light projected from the optical device onto a projection target are each captured by a camera; determines, based on the plurality of pieces of imaging data, whether or not there is at least one first structured light among the M pieces of structured light that has not been captured correctly; and, if it is determined that there is at least one first structured light, projects N (N is a natural number smaller than M) pieces of structured light including the at least one first structured light onto the projection target.

[0140] In the above aspect, if, as a result of capturing M pieces of structured light, there is at least one first structured light that has not been captured properly, N pieces of structured light that include the at least one first structured light and are smaller than M are projected, thereby making it possible to minimize the number of structured light pieces that are re-projected. As a result, even if there is structured light that has not been captured properly due to disturbances or the like, it is possible to optimize the number of projected structured light pieces.

[0141] (Appendix 10) A tenth aspect, which is a preferred example of the program of the present disclosure, causes a computer to perform the following steps: acquire a plurality of pieces of imaging data in which M (M is a natural number greater than or equal to 2) pieces of structured light projected from a projector onto a projection target are each captured by a camera; determine, based on the plurality of pieces of imaging data, whether or not there is at least one first structured light among the M pieces of structured light that has not been captured correctly; and, if it is determined that there is at least one first structured light, cause the projector to project N (N is a natural number smaller than M) pieces of structured light including the at least one first structured light onto the projection target.

[0142] In the above aspect, if, as a result of capturing M pieces of structured light, there is at least one first structured light that has not been captured properly, N pieces of structured light that include the at least one first structured light and are smaller than M are projected, thereby making it possible to minimize the number of structured light pieces that are re-projected. As a result, even if there is structured light that has not been captured properly due to disturbances or the like, it is possible to optimize the number of projected structured light pieces. [Explanation of symbols]

[0143] 10...projector, 10A...projector, 10B...projector, 11...storage device, 12...processing device (computer), 12a...projection control unit, 12b...imaging control unit, 12c...generation unit, 12d...generation unit, 12e...generation unit, 13...communication device, 14...image processing circuit, 15...optical device, 15a...light source, 15b...display panel, 15c...optical system, 16...operation device, 17...sensor, 20...camera, 30...terminal device, 31...display device, 100...system system, 100A...system, 100B...system, B1...button, B2...button, B3...button, B4...button, D0...imaging data, D0-1...imaging data, D1...first imaging data, D2...second imaging data, D3...change information, DC...correspondence information, DG0...structured light information, DG0-1...structured light information, DG1...first pattern information, DS1...first information, DS2...second information, G...projected image, G-1...first pattern, GM...image, GU...user interface Face image, G0...structured light, G1...first pattern, G2...second image, GS...structured light, GS-a...structured light, GS-b...structured light, GS-c...structured light, GS-d...structured light, GS1...first structured light, IMG...image data, PR1...program, PR2...program, PR3...program, RC...area, RP...area, RP1...projection range, Ra...area, Rb...area, Rc...area, Rd...message, Re...message, S10...step, S10 0...step, S110...step, S120...step, S120B...step, S130...step, S140...step, S20...step, S21...step, S22...step, S23...step, S30...step, S31...step, S32...step, S40...step, S50...step, S60...step, S70...step, S80...step, S90...step, S91...step, S92...step, SC...projection target.

Claims

1. acquiring a plurality of pieces of image data by capturing, with a camera, M (M is a natural number equal to or greater than 2) pieces of structured light projected from the projector onto the projection target; determining whether or not at least one first structured light beam is not properly captured among the M pieces of structured light beam based on the plurality of pieces of captured image data; When it is determined that the at least one first structured light is present, projecting N (N is a natural number smaller than M) structured lights including the at least one first structured light onto the projection target. A projection method characterized by:

2. and when it is determined that the number of repetitions of projecting the N structured lights is equal to or greater than a first number and that the at least one first structured light is present, projecting from the projector a message indicating that the at least one first structured light may be generated even if the projection of the N structured lights is performed. The projection method according to claim 1 .

3. and when it is determined that the number of repetitions of projecting the N structured lights is equal to or greater than a first number and that the at least one first structured light is present, projecting a message from the projector that includes one or more possible causes for the presence of the at least one first structured light. The projection method according to claim 1 .

4. and further comprising projecting from the projector a user interface image for setting the first number of times. The projection method according to claim 2 or 3.

5. projecting a first pattern of a uniform first color from the projector onto the projection target before sequentially projecting the M pieces of structured light from the projector onto the projection target; determining a projection range on the projection target onto which each of the M structured lights is projected based on the first pattern; determining whether or not the at least one first structured light is present is performed based on each of the M structured lights in the projection range; The projection method according to claim 1 .

6. acquiring a plurality of pieces of image data by capturing, with a camera, M (M is a natural number equal to or greater than 2) pieces of structured light projected from the projector onto the projection target; determining whether or not at least one first structured light beam is not properly captured among the M pieces of structured light beam based on the plurality of pieces of captured image data; determining whether or not one or both of the position and the attitude of the projector relative to the projection target has changed based on change information indicating a change in the one or both of the position and the attitude of the projector relative to the projection target; When it is determined that the at least one first structured light exists and one or both of the position and the attitude have not changed, projecting N (N is a natural number smaller than M) structured lights including the at least one first structured light onto the projection target; projecting the M structured lights onto the projection target when it is determined that the at least one first structured light exists and one or both of the position and the attitude has changed. A projection method characterized by:

7. Before sequentially projecting the M pieces of structured light from the projector onto the projection target, capturing an image of a first pattern of a uniform first color projected from the projector onto the projection target with the camera, thereby acquiring first captured data; After the M pieces of structured light are sequentially projected from the projector onto the projection target and before determining whether or not the at least one first structured light is present, capturing an image of the first pattern projected from the projector onto the projection target with the camera to acquire second captured data; The change information is identified based on the first imaging data and the second imaging data. The projection method according to claim 6.

8. acquiring first information indicating one or both of the attitude and the position before the M pieces of structured light are sequentially projected from the projector onto the projection target; acquiring second information indicating one or both of the attitude and the position after the M pieces of structured light are sequentially projected from the projector onto the projection target and before determining whether the at least one first structured light is present; The change information is identified based on the first information and the second information. The projection method according to claim 6.

9. an optical device; a processing device for controlling the operation of the optical device, The processing device includes: acquiring a plurality of pieces of image data in which M (M is a natural number equal to or greater than 2) pieces of structured light projected from the optical device onto a projection target are captured by a camera; determining whether or not at least one first structured light beam is not properly captured among the M pieces of structured light beam based on the plurality of pieces of captured image data; When it is determined that the at least one first structured light exists, projecting N (N is a natural number smaller than M) structured lights including the at least one first structured light onto the projection target. A projector characterized by:

10. acquiring a plurality of pieces of image data in which M (M is a natural number equal to or greater than 2) pieces of structured light projected from a projector onto a projection target are captured by a camera; determining whether or not at least one first structured light beam is not properly captured among the M pieces of structured light beam based on the plurality of pieces of captured image data; When it is determined that the at least one first structured light is present, the computer is caused to project N (N is a natural number smaller than M) pieces of structured light including the at least one first structured light onto the projection target. A program characterized by:

Images