Processing system, first processing unit, and second processing unit

Abstract

To provide a technology with which it is possible to appropriately identify the three-dimensional position of a device.SOLUTION: A first processing device comprises a light-emitting unit that emits light, a wave sending unit that emits a sound wave, and a camera. A second processing device comprises a first light reception unit that receives light, a first wave reception unit that receives a sound wave, and a first display unit that performs first display. The camera performs an image capturing process to capture the image of the first display. A first distance acquisition unit finds a first distance between the second processing device and the first processing device on the basis of the first timing difference between the first light reception timing of light by the first light reception unit and first wave reception timing of sound waves by the first wave reception unit. A first two-dimensional position identification unit identifies the first two-dimensional position of first display in the image obtained by the image capturing process. A first three-dimensional position identification unit identifies the first three-dimensional position of the second processing device on the basis of the first two-dimensional position and the first distance.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This disclosure relates to a technique for specifying the three-dimensional position of a device.

Background Art

[0002] Patent Document 1 discloses a technique for specifying the three-dimensional position of a person.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is room for improvement in the technique for specifying the three-dimensional position of a device.

[0005] Therefore, this disclosure has been made in view of the above points, and an object thereof is to provide a technique capable of appropriately specifying the three-dimensional position of a device.

Means for Solving the Problems

[0006] One embodiment of the processing system comprises a first processing unit and a second processing unit. The first processing unit includes a light-emitting unit that emits light, a sound-emitting unit that emits sound waves, and a camera. The second processing unit includes a first light-receiving unit that receives light, a first sound-receiving unit that receives sound waves, and a first display unit that performs a first display. The camera performs a shooting process to photograph the first display. The processing system comprises a first distance acquisition unit, a first two-dimensional position determination unit, and a first three-dimensional position determination unit. The first distance acquisition unit performs a distance acquisition process to determine a first distance between the second processing unit and the first processing unit based on a first timing difference between a first light-receiving timing of light at the first light-receiving unit and a first sound-receiving timing of sound waves at the first sound-receiving unit. The first two-dimensional position determination unit performs a two-dimensional position determination process to determine a first two-dimensional position of the first display in the image obtained by the shooting process. The first 3D positioning unit performs a 3D positioning process to determine the first 3D position of the second processing unit based on the first 2D position and the first distance.

[0007] Furthermore, one embodiment of the first processing device is the first processing device provided in the processing system described above.

[0008] Furthermore, one embodiment of the second processing device is a second processing device provided in the processing system described above. [Effects of the Invention]

[0009] The three-dimensional position of the second processing unit can be appropriately determined. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing an example of a processing system. [Figure 2] This is a schematic diagram showing an example of a processing unit. [Figure 3] This is a schematic diagram showing an example of a processing unit. [Figure 4] This is a schematic diagram showing an example of a control unit. [Figure 5] This is a schematic diagram showing an example of a control unit. [Figure 6]It is a schematic diagram for explaining an example of a method for specifying the two-dimensional position of a processing device in an image. [Figure 7] It is a schematic diagram for explaining an example of a method for specifying the three-dimensional position of a processing device. [Figure 8] It is a schematic diagram showing an example of the operation of a processing system. [Figure 9] It is a schematic diagram showing an example of a control unit. [Figure 10] It is a schematic diagram showing an example of a sensitivity adjustment process. [Figure 11] It is a schematic diagram showing an example of the operation of a processing system. [Figure 12] It is a schematic diagram showing an example of the operation of a processing system. [Figure 13] It is a schematic diagram showing an example of a control unit. [Figure 14] It is a schematic diagram showing an example of a processing system. [Figure 15] It is a schematic diagram showing an example of the display of a plurality of processing devices.

Embodiments for Carrying Out the Invention

[0011] FIG. 1 is a schematic diagram showing an example of a processing system 1. As shown in FIG. 1, the processing system 1 includes, for example, a processing device 2, a processing device 3, and a display device 10. The processing device 2 can control the display of the display device 10, for example. The display device 10 may be a liquid crystal display device or an organic EL (electro-luminescence) display device.

[0012]

[0013] ​The processing device 3 is, for example, a portable device. Therefore, the position of the processing device 3 changes. The user operates the processing device 3, for example, while holding the processing device 3 in the hand. The processing device 3 can receive a user operation (also referred to as a user operation). The processing device 3 transmits operation information indicating the received user operation to the processing device 2. The processing device 2 performs processing based on the received operation information. The processing device 2 controls, for example, the display of the display device 10 based on the operation information. Thereby, in the processing system 1, the display of the display device 10 changes according to the user operation on the processing device 3. The processing device 3 can be said to be, for example, a portable operation device 3 that receives user operations.

[0014] Also, in the processing system 1, the three-dimensional position of the processing device 3 whose position changes is specified by the cooperation of the processing device 2 and the processing device 3. Note that the processing system 1 may not include the display device 10.

[0015] FIG. 2 is a schematic diagram showing an example of the processing device 2. As shown in FIGS. 1 and 2, the processing device 2 includes, for example, a control unit 20, a storage unit 21, a light emitting unit 22, a wave transmitting unit 23, a camera 24, an interface 25, and an interface 26. These components are housed in a case 29 (see FIG. 1) provided in the processing device 2. The processing device 2 can also be said to be, for example, a processing circuit. The processing device 2 is, for example, a type of computer device.

[0016] Interface 25 is capable of communicating with the processing unit 3. Interface 25 can also be called, for example, an interface circuit. Alternatively, Interface 25 can also be called, for example, a communication unit or a communication circuit. Interface 25 may communicate with the processing unit 3 via wired communication or wireless communication. Interface 25 may communicate with the processing unit 3 in accordance with at least one communication standard. The at least one communication standard that Interface 25 conforms to may include at least one of Bluetooth®, USB (Universal Serial Bus), WiFi, and Ethernet. Interface 25 may also communicate with the processing unit 3 via the internet.

[0017] Interface 26 can communicate with external devices (also simply called external devices) of the processing system 1, for example. External devices may include, for example, the display device 10. Interface 26 can also be called, for example, an interface circuit. Interface 26 can also be called, for example, a communication unit or communication circuit. Interface 26 transmits, for example, an image signal generated by the control unit 20 to the display device 10. The display device 10 displays an image based on the image signal received from the processing unit 2. Interface 26 may conform to HDMI (High-Definition Multimedia Interface) (registered trademark), DisplayPort, or other communication standards. Interface 26 may also communicate with devices other than the display device 10.

[0018] The light-emitting unit 22 is capable of emitting light 220 to the outside of the case 29. The light 220 is, for example, infrared light. In this case, the light-emitting unit 22 can be called an infrared light-emitting unit. The light 220 may also be visible light. The light-emitting unit 22 may be composed of, for example, an LED (Light Emitting Diode), or it may be composed of other components. The light-emitting unit 22 can also be called, for example, a light-emitting circuit.

[0019] The transmitting unit 23 is capable of emitting sound waves 230 to the outside of the case 29. The transmitting unit 23 can also be called, for example, a speaker. The sound waves 230 are, for example, ultrasonic waves. In this case, the transmitting unit 23 can also be called an ultrasonic transmitting unit, an ultrasonic transmitting unit, or an ultrasonic speaker. The sound waves 230 may be sound waves in the audible frequency range, that is, sound waves that humans can hear. The transmitting unit 23 may be equipped with, for example, a piezoelectric element or other components. The transmitting unit 23 may be composed of a MEMS (Micro Electro Mechanical System).

[0020] Camera 24 is capable of photographing the processing unit 3. Camera 24 performs a photographic process to photograph the display of the display unit 34, which is provided by the processing unit 3, as described later. The image 240 obtained through the photographic process will show the display of the display unit 34. In the photographic process, things other than the display of the display unit 34 may also be photographed.

[0021] As will be described later, the display unit 34 is composed of, for example, an infrared light emitter and displays using infrared light. In this case, the camera 24 is, for example, an infrared camera and can detect the infrared light emitted by the display unit 34. In other words, the camera 24 can capture the infrared display performed by the display unit 34. The camera 24 may include, for example, an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor and a lens. The image 240 obtained by the camera 24's shooting process is stored as image data in the storage unit 21 via the control unit 20.

[0022] The control unit 20 can comprehensively manage the operation of the processing unit 2 by controlling other components of the processing unit 2. The control unit 20 can also be called a control circuit, for example. The control unit 20 includes, for example, at least one processor. The control unit 20 may include, for example, a CPU (Central Processing Unit).

[0023] The storage unit 21 may include non-temporary recording media that can be read by the CPU of the control unit 20, such as ROM (Read Only Memory) and RAM (Random Access Memory). The storage unit 21 stores, for example, a program 210 for controlling the processing unit 2. Various functions of the control unit 20 are realized, for example, by the CPU of the control unit 20 executing the program 210 in the storage unit 21.

[0024] The configuration of the control unit 20 is not limited to the example above. For example, the control unit 20 may have multiple CPUs. The control unit 20 may also have at least one DSP (Digital Signal Processor). Furthermore, all or some of the functions of the control unit 20 may be implemented by hardware circuits that do not require software to implement those functions. In addition, the storage unit 21 may have a computer-readable, non-temporary recording medium other than ROM and RAM. The storage unit 21 may have, for example, a small hard disk drive and an SSD (Solid State Drive).

[0025] Figure 3 is a schematic diagram showing an example of a processing unit 3. As shown in Figures 1 and 3, the processing unit 3 includes, for example, a control unit 30, a storage unit 31, a light receiving unit 32, a wave receiving unit 33, a display unit 34, an operation unit 35, an inertial sensor 36, and an interface 37. These components are housed in a case 39 (see Figure 1) of the processing unit 3. The processing unit 3 can also be called, for example, a processing circuit. The processing unit 3 is, for example, a type of computer device.

[0026] Interface 37 can communicate with interface 25 of processing unit 2. Interface 37 can also be called, for example, an interface circuit, a communication unit, or a communication circuit. Interface 37 has a configuration similar to interface 25, for example.

[0027] The light-receiving unit 32 is capable of receiving light 220 emitted by the light-emitting unit 22 of the processing unit 2. The light-receiving unit 32 is, for example, an infrared light-receiving unit. The light-receiving unit 32 may be composed of, for example, a photodiode, or other components. The light-receiving unit 32 can also be called, for example, a light-receiving circuit.

[0028] The wave receiving unit 33 is capable of receiving sound waves 230 emitted by the wave transmitting unit 23 of the processing unit 2. The wave receiving unit 33 is, for example, an ultrasonic microphone. An ultrasonic microphone can also be called an ultrasonic receiver or ultrasonic wave receiving unit. The wave receiving unit 33 may be equipped with, for example, a piezoelectric element or other components. The wave receiving unit 33 may be made of MEMS.

[0029] The display unit 34 can, for example, display a predetermined information. The display unit 34 is, for example, an infrared light-emitting unit and can display information using infrared light. The display unit 34 may, for example, consist of at least one infrared light-emitting element. The infrared light-emitting element may be, for example, an LED. The display unit 34 can also be called, for example, a display circuit.

[0030] The display unit 34 may display a predetermined pattern. The predetermined pattern may be, for example, a dot pattern composed of multiple dots, or it may be any other pattern. If the predetermined pattern is a dot pattern, the display unit 34 may be composed of multiple infrared light-emitting elements, for example, each of which displays multiple dots using infrared light. In other words, the display unit 34 displays the predetermined pattern using infrared light. The display unit 34 may also display using visible light. In this case, the display unit 34 may be composed of, for example, a visible light-emitting unit having at least one visible light-emitting element. The visible light-emitting element may be an LED.

[0031] The operation unit 35 is capable of receiving user input. The operation unit 35 outputs operation information indicating the received user input to the control unit 30. The operation unit 35 may have, for example, at least one operation button. At least one operation button may include a hardware button. The operation unit 35 may be equipped with a touch sensor to detect user touch input. The operation unit 35 may also be equipped with a display and a touch sensor to detect user touch input on the display screen. In this case, software buttons as operation buttons may be displayed on the display.

[0032] The inertial sensor 36 includes, for example, a gyro sensor for detecting the angular velocity of the processing unit 3 and an accelerometer for detecting the acceleration of the processing unit 3. The gyro sensor of the inertial sensor 36 is, for example, a 3-axis gyro sensor. The accelerometer of the inertial sensor 36 is, for example, a 3-axis accelerometer. The inertial sensor 36 is also called, for example, an IMU (Inertial Measurement Unit) sensor. The detection results of the inertial sensor 36, that is, the detection results of the gyro sensor and the accelerometer, are input to the control unit 30.

[0033] The control unit 30 can comprehensively manage the operation of the processing unit 3 by controlling other components of the processing unit 3. The control unit 30 can also be called a control circuit, for example. The control unit 30 includes, for example, at least one processor. The control unit 30 may also include, for example, a CPU.

[0034] The control unit 30 transmits operation information from the operation unit 35 to the processing unit 2 via the interface 37. The processing unit 2 receives the operation information from the interface 25 and inputs it to the control unit 20. The control unit 20 controls the display of the display device 10 according to the operation information via the interface 26. For example, the control unit 20 may enlarge, reduce, or scroll the display of the display device 10 according to the operation information. The control unit 20 may also change the position of the displayed object on the display screen of the display device 10 according to the operation information.

[0035] The storage unit 31 may include non-temporary recording media such as ROM and RAM that can be read by the CPU of the control unit 30. The storage unit 31 stores, for example, a program 310 for controlling the processing unit 3. Various functions of the control unit 30 are realized, for example, by the CPU of the control unit 30 executing the program 310 in the storage unit 31.

[0036] The configuration of the control unit 30 is not limited to the example described above. For example, the control unit 30 may have multiple CPUs or at least one DSP. Furthermore, all or some of the functions of the control unit 30 may be implemented by hardware circuits that do not require software to perform those functions. Also, the storage unit 31, like the storage unit 21, may have a computer-readable, non-temporary recording medium other than ROM and RAM.

[0037] Figure 4 is a schematic diagram showing an example of multiple functional blocks formed in the control unit 20 of the processing unit 2 by the execution of the program 210. The control unit 20 includes, for example, a light emission control unit 200, a wave transmission control unit 202, a camera control unit 204, a two-dimensional positioning unit 206, and a three-dimensional positioning unit 208 as functional blocks.

[0038] Furthermore, all or some of the functions of the light emission control unit 200 may be implemented by hardware circuits that do not require software to realize those functions. The same applies to the wave transmission control unit 202, camera control unit 204, 2D positioning unit 206, and 3D positioning unit 208.

[0039] The light emission control unit 200 can control the light emission of the light emission unit 22. For example, the light emission control unit 200 can cause the light emission unit 22 to emit light or to stop the light emission of the light emission unit 22. The wave transmission control unit 202 can control the wave transmission of the wave transmission unit 23. For example, the wave transmission control unit 202 can cause the wave transmission unit 23 to transmit waves or to stop the wave transmission unit 23. The camera control unit 204 can control the shooting of the camera 24. The camera control unit 204 can cause the camera 24 to take pictures or to stop the camera 24 from taking pictures.

[0040] The 2D positioning unit 206 performs a 2D positioning process to determine the 2D position of the display unit 34 in the image 240 obtained by the camera 24. The 3D positioning unit 208 performs a 3D positioning process to determine the 3D position of the processing unit 3 based on the 2D position determined in the 2D positioning process (also called the determined 2D position 207) and the acquired distance 303 transmitted from the processing unit 3, which will be described later. The acquired distance 303 is the distance between the processing unit 3 and the processing unit 2, as determined by the processing unit 3. Hereafter, the 3D position determined in the 3D positioning process may be referred to as the determined 3D position 209. The 2D positioning process and the 3D positioning process will be described in detail later.

[0041] The control unit 20 performs processing based on a specific 3D position 209 identified by the 3D position identification process. The control unit 20 may, for example, control the display of the display device 10 according to the specific 3D position 209 through the interface 26. The control unit 20 may enlarge, reduce, or scroll the display of the display device 10 according to the specific 3D position 209. The control unit 20 may also change the position of the object to be displayed on the display screen of the display device 10 according to the specific 3D position 209.

[0042] Figure 5 is a schematic diagram showing an example of multiple functional blocks formed in the control unit 30 of the processing unit 3 by the execution of the program 310. The control unit 30 includes, for example, a display control unit 300, a distance acquisition unit 302, and a rotation identification unit 304 as functional blocks. Note that all or some of the functions of the display control unit 300 may be implemented by hardware circuits that do not require software to realize those functions. The same applies to the distance acquisition unit 302 and the rotation identification unit 304.

[0043] The display control unit 300 can control the display of the display unit 34. For example, the display control unit 300 can cause the display unit 34 to display something or to stop the display of the display unit 34. If the display unit 34 is composed of a light-emitting unit, the display control unit 300 can cause the display unit 34 to emit light or to stop the light-emitting unit. The display control unit 300 can also be said to be a light-emitting control unit.

[0044] The distance acquisition unit 302 performs distance acquisition processing to determine the distance between the processing unit 3 and the processing unit 2. In the distance acquisition processing, the distance between the processing unit 3 and the processing unit 2 is determined based on the difference between the timing of receiving light 220 at the light receiving unit 32 and the timing of receiving sound waves 230 at the wave receiving unit 33. In the processing system 1, for example, the distance between the processing unit 3 and the processing unit 2 varies within a range of a few meters or less (for example, within a range of 5 meters or less). The distance determined by the distance acquisition unit 302 (also called the acquired distance 303) is transmitted to the processing unit 2 through the interface 37. In the processing unit 2, the acquired distance 303 received at the interface 25 is stored in the storage unit 21 through the control unit 20. The distance acquisition processing will be described in detail later.

[0045] The rotation identification unit 304 can identify the rotation of the processing unit 3 based on the detection results of the inertial sensor 36. The rotation identification unit 304 can, for example, identify the rotation of the processing unit 3 around each of the three mutually orthogonal axes. The rotation identification unit 304 generates rotation information indicating the result of identifying the rotation of the processing unit 3. The rotation information is transmitted to the processing unit 2, for example, through the interface 37. In the processing unit 2, the rotation information received at the interface 25 is stored in the storage unit 21 through the control unit 20. The control unit 20 performs processing based on the rotation information in the storage unit 21, for example. The control unit 20 may enlarge, reduce, or scroll the display on the display device 10 according to the rotation information. The control unit 20 may also change the orientation of the displayed object on the display screen of the display device 10 according to the rotation information.

[0046] <Example of distance acquisition process> In the processing unit 2, the emission of light from the light-emitting unit 22 and the transmission of sound waves from the transmission unit 23 occur simultaneously. In other words, the timing of the emission of light 220 from the light-emitting unit 22 and the transmission timing of sound waves 230 from the transmission unit 23 are set to be the same. Then, the distance acquisition unit 302 of the processing unit 3 calculates the difference (also called the arrival timing difference) between the timing of the light 220 being received by the light-receiving unit 32 and the timing of the sound waves 230 being received by the wave-receiving unit 33.

[0047] The timing of light reception 220 at the light receiving unit 32 can also be said to be the start timing of light reception 220 at the light receiving unit 32. The timing of sound wave reception 230 at the wave receiving unit 33 can also be said to be the start timing of sound wave reception 230 at the wave receiving unit 33. The timing of light emission 220 at the light emitting unit 22 can also be said to be the start timing of light emission 220 at the light emitting unit 22. The timing of sound wave transmission 230 at the wave transmitting unit 23 can also be said to be the start timing of sound wave transmission 230 at the wave transmitting unit 23.

[0048] Here, the speed of light 220 is greater than the speed of sound wave 230. Therefore, when the light-emitting unit 22 emits light and the transmitting unit 23 transmits waves simultaneously, the light-receiving unit 32 receives the light 220, and then the receiving unit 33 receives the sound wave 230. On the other hand, because the speed of light 220 is much greater than the speed of sound wave 230, when the light-emitting unit 22 emits light and the transmitting unit 23 transmits waves simultaneously, by the time the light 220 reaches the light-receiving unit 32, the sound wave 230 has hardly traveled from the transmitting unit 23. Therefore, when the light-emitting unit 22 emits light and the transmitting unit 23 transmits waves simultaneously, the timing of the light 220's reception at the light-receiving unit 32 almost coincides with the timing of the sound wave 230's transmission at the transmitting unit 23, and the reception timing can be considered the same as the transmission timing.

[0049] Therefore, the distance acquisition unit 302 measures the time from the timing of receiving the light 220 at the light receiving unit 32 to the timing of receiving the sound wave 230 at the wave receiving unit 33 as the arrival timing difference. The arrival timing difference can be considered as the time from the timing of transmitting the sound wave 230 at the wave transmitting unit 23 to the timing of receiving the sound wave 230 at the wave receiving unit 33. The processing unit 3 is equipped with, for example, a clock for measuring time. The distance acquisition unit 302 can determine the arrival timing difference based on the output of the clock.

[0050] The distance acquisition unit 302 calculates the distance between the processing unit 3 and the processing unit 2 based on the calculated arrival timing difference. For example, the distance acquisition unit 302 takes the value obtained by multiplying the arrival timing difference by the speed of the sound wave 230 as the acquired distance 303. If the arrival timing difference is, for example, 0.01 seconds and the speed of the sound wave 230 is 340 m / s, then the acquired distance 303 will be 3.4 m. The acquired distance 303 represents the straight-line distance between the processing unit 3 and the processing unit 2.

[0051] In this way, the distance acquisition unit 302 can appropriately determine the distance between the processing unit 3 and the processing unit 2 based on the timing difference between the timing of receiving the light 220 and the timing of receiving the sound wave 230.

[0052] Hereafter, when we simply refer to the light reception timing, we mean the timing of the light 220 being received by the light receiving unit 32. Similarly, when we simply refer to the wave reception timing, we mean the timing of the sound wave 230 being received by the wave receiving unit 33. Furthermore, when we simply refer to the light emission timing, we mean the timing of the light 220 being emitted by the light emitting unit 22. Furthermore, when we simply refer to the wave transmission timing, we mean the timing of the sound wave 230 being transmitted by the wave transmitting unit 23. Also, when we simply refer to the light reception, we mean the light 220 being received by the light receiving unit 32, and when we simply refer to the wave reception, we mean the sound wave 230 being received by the wave receiving unit 33. Furthermore, when we simply refer to the light emission, we mean the light 220 being emitted by the light emitting unit 22, and when we simply refer to the wave transmission, we mean the sound wave 230 being transmitted by the wave transmitting unit 23.

[0053] As can be understood from the above explanation, the distance acquisition process assumes that the light reception timing can be considered to be the same as the transmission timing, and the distance between the processing unit 3 and the processing unit 2 is determined accordingly. Therefore, if the light reception timing deviates significantly from the transmission timing, the accuracy of the acquired distance 303 decreases. If the light emission timing and the transmission timing are significantly out of sync, the light reception timing will also deviate significantly from the transmission timing, making it difficult to consider the light reception timing as being the same as the transmission timing. In other words, if the light emission timing and the transmission timing are significantly out of sync, the light reception timing will deviate from the transmission timing to such an extent that the accuracy of the acquired distance 303 will no longer meet the required accuracy. In this disclosure, the simultaneous emission of light from the light emission unit 22 and the transmission of waves from the transmission unit 23 includes not only the case where the light emission timing and the transmission timing are perfectly aligned, but also the case where the light emission timing and the transmission timing are out of sync to such an extent that the light reception timing can be considered to be the same as the transmission timing. In other words, the simultaneous emission of light from the light-emitting unit 22 and the transmission of waves from the transmission unit 23 includes cases where the timing of light emission and the timing of wave transmission are out of sync, as long as the accuracy of the acquisition distance 303 meets the required accuracy. The required accuracy for the acquisition distance 303 is set according to the intended use of the acquisition distance 303.

[0054] <An example of 2D positioning processing> The 2D position identified by the 2D positioning process is represented by coordinate information in a 2D Cartesian coordinate system 1000 set for the image 240 obtained by the camera 24. Figure 6 is a schematic diagram showing an example of a 2D Cartesian coordinate system 1000 (also simply called coordinate system 1000). The origin 1001 of coordinate system 1000 is set, for example, at the center of image 240. The X-axis of coordinate system 1000 is set, for example, along one side of image 240. The Y-axis of coordinate system 1000 is set to be orthogonal to the X-axis of coordinate system 1000. The position of each pixel constituting image 240 is represented by the X-axis value and Y-axis value in coordinate system 1000.

[0055] Image 240 shows the display 340 of the display unit 34. In the example in Figure 6, the display 340 is a dot pattern composed of three dots arranged in a row. The three dots are formed, for example, by the light emitted by three light-emitting elements.

[0056] When the two-dimensional positioning unit 206 performs two-dimensional positioning processing, it reads the image 240 from the storage unit 21. The two-dimensional positioning unit 206 then identifies the X coordinate value X1 and Y coordinate value Y1 of the display 340 in the read image 240 as the two-dimensional position of the display 340. If the display 340 is captured across multiple pixels in the image 240, the two-dimensional positioning unit 206 may use the X coordinate value X1 and Y coordinate value Y1 of the display 340 as in the image 240.

[0057] <An example of 3D positioning processing> The 3D position identified by the 3D positioning process is represented by coordinate information in a 3D Cartesian coordinate system 1100, which is set to include, for example, the shooting range 245 of the camera 24. Figure 7 is a schematic diagram showing an example of a 3D Cartesian coordinate system 1100 (also simply called coordinate system 1100).

[0058] The origin 1101 of coordinate system 1100 is set, for example, on camera 24. The origin 1101 may be set, for example, on the lens of camera 24, on the image sensor of camera 24, or on another location. The z-axis of coordinate system 1100 is set, for example, parallel to the optical axis direction of camera 24, with the positive side pointing towards the back of the shooting range 245. The x-axis of coordinate system 1100 is set perpendicular to the z-axis of coordinate system 1100, and the y-axis of coordinate system 1100 is set perpendicular to the x-axis and z-axis of coordinate system 1100. The three-dimensional position of processing unit 3 is represented by the x-coordinate, y-coordinate, and z-coordinate values ​​of coordinate system 1100.

[0059] The 3D positioning unit 208 identifies the direction 1110 of the processing unit 3 as seen from the processing unit 2, based on the specific 2D position 207 identified by the 2D positioning unit 206. Based on the specific 2D position 207, the 3D positioning unit 208 identifies, for example, the direction of the display 340 of the display unit 34 as seen from the camera 24 as the direction 1110 of the processing unit 3. The 3D positioning unit 208 can identify the direction of the display 340 as seen from the camera 24, for example, based on the specific 2D position 207 and the field of view and resolution (in other words, the number of pixels) of the camera 24. Then, based on the identified direction 1110 and the acquired distance 303 obtained by the processing unit 3, the 3D positioning unit 208 identifies the 3D position of the processing unit 3 in terms of x, y, and z coordinate values ​​in the coordinate system 1100.

[0060] As described above, the processing system 1 identifies the three-dimensional position of the processing unit 3. By repeatedly identifying the three-dimensional position of the processing unit 3, the translational motion of the processing unit 3 in the x-axis, y-axis, and z-axis directions can be identified. Furthermore, the processing system 1 also identifies the rotation of the processing unit 3 around the three axes. This enables the processing system 1 to implement a 6DoF (Degrees of Freedom) controller that detects the movement of the processing unit 3.

[0061] The rotation identification unit 304, which identifies the rotation of the processing unit 3, may be provided in the processing unit 2 instead of the processing unit 3. In this case, the detection result of the inertial sensor 36 of the processing unit 3 is transmitted to the processing unit 2 via the interface 37. In the processing unit 2, the control unit 20 may function as the rotation identification unit 304 that identifies the rotation of the processing unit 3 based on the detection result received via the interface 25. Furthermore, the rotation of the processing unit 3 does not need to be identified in the processing system 1. Also, the processing unit 3 does not need to have an operation unit 35 or an inertial sensor 36.

[0062] <Example of processing system operation> Figure 8 is a schematic diagram showing an example of the positioning process performed in processing system 1. In the positioning process, processing unit 2 and processing unit 3 work together to determine the three-dimensional position of processing unit 3.

[0063] In the position determination process, first, in the processing unit 2, the emission of light 220 by the light-emitting unit 22 and the transmission of sound waves 230 by the transmission unit 23 are performed simultaneously (step s1). After step s1, in the processing unit 3, the light-receiving unit 32 receives the light 220, and then the wave-receiving unit 33 receives the sound waves 230 (step s11). Next, in step s12, distance acquisition processing is performed to determine the distance between the processing unit 3 and the processing unit 2. Hereafter, the processes in steps s1, s11, and s12 are sometimes collectively referred to as the distance measurement process. In the distance measurement process, the processing unit 2 emits light 220 and transmits sound waves 230, and the processing unit 3 receives the light 220 and sound waves 230. Then, in the distance measurement process, the processing unit 3 determines the distance between the processing unit 3 and the processing unit 2 based on the difference between the timing of receiving the light 220 and the timing of receiving the sound waves 230.

[0064] After the distance measurement process, in step s13, the processing unit 3 receives the acquired distance 303 from the interface 37 and displays the information 340 on the display unit 34.

[0065] After step s13, in step s2, the interface 25 of the processing unit 2 receives the acquired distance 303. The received acquired distance 303 is stored in the storage unit 21. Next, in step s3, the camera 24 performs the shooting process. Next, in step s4, a two-dimensional positioning process is performed to determine the two-dimensional position of the display 340 of the display unit 34. Next, in step s5, a three-dimensional positioning process is performed to determine the three-dimensional position of the processing unit 3.

[0066] In processing system 1, the above position identification process is repeatedly executed, thereby repeatedly determining the three-dimensional position of processing unit 3. Processing unit 2 controls the display of, for example, the display device 10 according to the repeatedly obtained specific three-dimensional position 209.

[0067] The distance acquisition unit 302 may be provided in the processing unit 2 instead of the processing unit 3. In this case, for example, the control unit 30 of the processing unit 3 identifies the arrival timing difference and notifies the processing unit 2 of the identified arrival timing difference through the interface 37. In the processing unit 2, the control unit 20 may function as the distance acquisition unit 302 that calculates the distance between the processing unit 3 and the processing unit 2 based on the arrival timing difference notified by the processing unit 2. The acquired distance 303 obtained by the control unit 20 is stored in the storage unit 21. The 3D position identification unit 208 reads the acquired distance 303 from the storage unit 21 and uses it.

[0068] Furthermore, the two-dimensional positioning unit 206 may be provided in the processing unit 3 instead of the processing unit 2. In this case, for example, the control unit 20 of the processing unit 2 transmits the image 240 obtained by the camera 24 to the processing unit 3 through the interface 25. In the processing unit 3, the control unit 20 may function as a two-dimensional positioning unit 206 that performs two-dimensional positioning processing using the image 240 received through the interface 25. The specific two-dimensional position 207 identified by the control unit 20 is transmitted to the processing unit 2.

[0069] Furthermore, the 3D positioning unit 208 may be provided in the processing unit 3 instead of the processing unit 2. In this case, for example, the control unit 30 of the processing unit 3 may function as a 3D positioning unit 208 that identifies the 3D position of the processing unit 3 based on a specific 2D position 207 identified by a 2D positioning process performed in the processing unit 2 or the processing unit 3, and an acquired distance 303 obtained by a distance acquisition process performed in the processing unit 2 or the processing unit 3. The specific 3D position 209 identified by the control unit 30 may be transmitted to the processing unit 2.

[0070] As described above, in the processing system 1, the distance between the processing unit 3 and the processing unit 2 is determined based on the difference between the timing of light 220 reception at the light receiving unit 32 and the timing of sound wave 230 reception at the wave receiving unit 33. This allows the distance between the processing unit 3 and the processing unit 2 to be appropriately determined. Therefore, the three-dimensional position of the processing unit 3 can be appropriately identified based on the appropriately determined distance.

[0071] Furthermore, as shown in the example in Figure 5, if the processing device 3 has a distance acquisition unit 302, the processing load on the processing device 2 can be reduced.

[0072] Furthermore, as shown in the example in Figure 4, if the processing unit 2 that acquires the image 240 has a two-dimensional positioning unit 206, the processing unit 2 can perform two-dimensional positioning processing using the image 240 it has acquired. This eliminates the need, for example, to transfer the image 240 between the processing unit 2 and the processing unit 3 for two-dimensional positioning processing. Therefore, the processing load on the processing units 2 and 3 can be reduced. The image 240 may also be transmitted from the processing unit 2 to the processing unit 3.

[0073] Furthermore, as shown in the example in Figure 4, if the processing unit 2 that performs processing based on a specific three-dimensional position 209 has a three-dimensional position identification unit 208, then, for example, there is no need to transfer the specific three-dimensional position 209 between the processing unit 2 and the processing unit 3. Therefore, the processing load on both the processing unit 2 and the processing unit 3 can be reduced. Note that the specific three-dimensional position 209 may also be transmitted from the processing unit 2 to the processing unit 3.

[0074] Furthermore, if the light 220 emitted by the light-emitting unit 22 is infrared light, the light 220 will be less visible to the user of the processing system 1. Therefore, for example, it will be less likely that the user will feel annoyed by the visibility of the light 220.

[0075] Furthermore, if the sound waves 230 emitted by the transmitter 23 are ultrasonic waves, the sound waves 230 become less audible to the user. Therefore, for example, it becomes less likely that the user will feel bothered by hearing the sound waves 230.

[0076] Furthermore, if the display unit 34 is an infrared light emitter, the display 340 of the display unit 34 becomes less visible to the user. Therefore, for example, it becomes less likely that the user will feel annoyed by the visibility of the display 340. Note that the display 340 of the display unit 34 may be visible light as described above. Also, the display 340 may be a non-electrical display (in other words, a display that does not require power to be realized), such as a display by printing, painting, or engraving. If the display 340 is a non-electrical display, the display control unit 300 becomes unnecessary.

[0077] <Other examples of processing systems> <Another example from the first point> Figure 9 is a schematic diagram showing another example of the configuration of the control unit 30 of the processing unit 3. In the example of Figure 9, the control unit 30 further includes a sensitivity adjustment unit 306 as a functional block for adjusting the reception sensitivity of the reception unit 33. Note that all or some of the functions of the sensitivity adjustment unit 306 may be implemented by hardware circuits that do not require software to realize those functions. Hereafter, when simply referred to as reception sensitivity, it means the reception sensitivity of the reception unit 33.

[0078] The wave receiving unit 33 has, for example, an amplifier that amplifies the sound wave 230. The sensitivity adjustment unit 306 can adjust the wave receiving sensitivity by adjusting the gain of the amplifier in the wave receiving unit 33. If the amplifier gain is increased, the wave receiving sensitivity increases, and if the amplifier gain is decreased, the wave receiving sensitivity decreases.

[0079] Figure 10 is a schematic diagram showing an example of the sensitivity adjustment process performed by the sensitivity adjustment unit 306. Figure 10 also shows a graph illustrating how the reception sensitivity changes over time due to the sensitivity adjustment process. The horizontal and vertical axes of the graph represent time and reception sensitivity, respectively.

[0080] When the distance between the processing unit 3 and the processing unit 2 is small, the reception intensity of the sound wave 230 at the receiving unit 33 increases. In this case, if the reception sensitivity is high, the receiving unit 33 may not be able to properly receive the sound wave 230. On the other hand, when the distance between the processing unit 3 and the processing unit 2 is large, the reception intensity of the sound wave 230 at the receiving unit 33 decreases. In this case, if the reception sensitivity is low, the receiving unit 33 may not be able to properly receive the sound wave 230.

[0081] Therefore, as shown in Figure 10, the sensitivity adjustment unit 306 performs a sensitivity adjustment process to increase the reception sensitivity in proportion to the elapsed time from the light reception timing T1 to the wave reception timing T2. The longer the time from the light reception timing T1 to the wave reception timing T2, the greater the distance between the processing unit 3 and the processing unit 2. Thus, by increasing the reception sensitivity in proportion to the elapsed time from the light reception timing T1 to the wave reception timing T2, the reception sensitivity increases as the distance between the processing unit 3 and the processing unit 2 increases. In other words, the smaller the distance between the processing unit 3 and the processing unit 2, the smaller the reception sensitivity. As a result, the reception sensitivity is appropriately adjusted according to the distance between the processing unit 3 and the processing unit 2, and the reception unit 33 can appropriately receive the sound waves 230 regardless of the size of the distance.

[0082] The sensitivity adjustment unit 306 sets the reception sensitivity to an initial value when the light receiving unit 32 and the wave receiving unit 33 are not receiving light 220 and sound waves 230, respectively. The initial value is set to a reception sensitivity such that the wave receiving unit 33 can properly receive sound waves 230 when the processing unit 3 is closest to the processing unit 2. Then, when the light receiving unit 32 receives light 220, the sensitivity adjustment unit 306 gradually increases the reception sensitivity. At this time, the sensitivity adjustment unit 306 may increase the reception sensitivity continuously as in the example in Figure 10, or it may increase the reception sensitivity in a stepwise manner. The sensitivity adjustment unit 306 gradually increases the reception sensitivity until the wave receiving unit 33 receives sound waves 230. When the wave receiving unit 33 receives sound waves 230, the sensitivity adjustment unit 306 returns the reception sensitivity to an initial value. Subsequently, when the light receiving unit 32 receives light 220, the sensitivity adjustment unit 306 gradually increases the reception sensitivity again from the initial value. After that, the sensitivity adjustment unit 306 operates in the same manner.

[0083] <Another example, part 2> In the positioning process shown in the example in Figure 8, the imaging process is performed after the distance measurement process, but the distance measurement process and the imaging process may be executed in parallel. This shortens the time from when the positioning process starts until the 3D position of the processing unit 3 is determined. In the case of parallel execution of the distance measurement process and the imaging process, it is sufficient that at least a portion of the execution period of the distance measurement process and at least a portion of the execution period of the imaging process overlap.

[0084] When the distance measurement process and the image capture process are executed in parallel, in the position determination process, first, the control unit 20 of the processing unit 2 causes the camera 24 to start the image capture process, and also causes the light-emitting unit 22 and the wave-transmitting unit 23 to emit light and transmit waves, respectively. At this time, the display unit 34 of the processing unit 3 is either constantly displaying or intermittently displaying at very short intervals. The image capture process may start simultaneously with the light emission and wave transmission, before the light emission and wave transmission, or after the light emission and wave transmission. While the image capture process is being executed, light reception and wave reception are performed, and the distance acquisition process is executed. The distance acquisition process may end simultaneously with the image capture process, before the image capture process ends, or after the image capture process ends. Once the image capture process is completed and the image 240 is obtained, the obtained image 240 is used to perform the 2D position determination process. Then, the specific 2D position 207 determined in the 2D position determination process and the acquired distance 303 obtained in the distance acquisition process are used to perform the 3D position determination process.

[0085] Thus, even when the distance measurement and imaging processes are executed in parallel, if the next parallel execution of the distance measurement and imaging processes begins after the 2D and 3D position determination processes have been executed, the interval for determining the 3D position of the processing unit 3 becomes long. As a result, for example, it becomes difficult to precisely determine the translational motion of the processing unit 3.

[0086] Therefore, the 3D position of the processing unit 3 may be repeatedly determined by pipeline processing of the image capture process, the 2D position determination process, and the 3D position determination process, as well as by pipeline processing of the distance measurement process and the 3D position determination process. This allows the image capture process, distance measurement process, 2D position determination process, and 3D position determination process to be executed in an assembly-line manner, and the interval for determining the 3D position of the processing unit 3 can be shortened. As a result, for example, the translational motion of the processing unit 3 can be determined in detail. Figure 11 is a schematic diagram showing an example of the operation of the processing system 1 in this case.

[0087] Figure 11 shows a period of 250 (also called a shooting frame period 250 or frame period 250) during which the camera 24 performs the shooting process once. The camera 24 repeatedly performs the shooting process at intervals of the length of the shooting frame period 250. Figure 11 shows four consecutive shooting frame periods 250A, 250B, 250C, and 250D. In the example in Figure 11, the shooting process is performed for the first time in the initial shooting frame period 250A, and then the shooting process is repeated thereafter. If the shooting frame rate of the camera 24 is 60fps (frames per second), the length of the shooting frame period 250 is approximately 16.667ms. Figure 12, which will be described later, shows six consecutive frame periods 250A, 250B, 250C, 250D, 250E, and 250F.

[0088] Figure 11 shows the relationship between multiple frame periods 250 and the execution periods of various processes performed by the processing system 1. In Figure 11, the execution periods of the image capture process, 2D positioning process, distance measurement process, and 3D positioning process are shown as rectangles. Also in Figure 11, the execution periods of the image transmission and storage process and the distance transmission and storage process are also shown as rectangles. The image transmission and storage process refers to the process from when the camera 24 starts transmitting the image 240 until the image 240 is stored in the storage unit 21. The distance transmission and storage process refers to the process from when the acquired distance 303 is obtained in the processing unit 3 until the acquired distance 303 is transmitted to the processing unit 2 and stored in the storage unit 21.

[0089] In the example shown in Figure 11, the image capture process and the distance measurement process are executed in parallel during each frame period of 250. Hereafter, the pair of image capture and distance measurement processes executed in parallel during frame period 250x will be referred to as the image capture process 600x and the distance measurement process 630x, respectively. x is a variable, where x = A, B, C, D... The image 240 obtained in the image capture process 600x will be referred to as image 240x, and the image transmission and storage process for image 240x will be referred to as the image transmission and storage process 610x. The two-dimensional position determination process using image 240x will be referred to as the two-dimensional position determination process 620x, and the specific two-dimensional position 207 determined in the two-dimensional position determination process 620x will be referred to as the specific two-dimensional position 207x. The acquired distance 303 obtained in the distance measurement process 630x will be referred to as the acquired distance 303x, and the distance transmission and storage process for the acquired distance 303x will be referred to as the distance transmission and storage process 640x. The 3D position determination process that uses the acquired distance 303x and the specific 2D position 207x is called the 3D position determination process 650x. In Figure 11, the last letter of the code for each process is shown in the rectangle representing the execution period of the shooting process 600x, image transmission and storage process 610x, 2D position determination process 620x, distance measurement process 630x, distance transmission and storage process 640x, and the last letter of the code for each process.

[0090] In the following explanation, the image capture process 600x, image transmission and storage process 610x, 2D position determination process 620x, distance measurement process 630x, distance transmission and storage process 640x, and 3D position determination process 650x for the same frame period 250x are assumed to be corresponding processes. Therefore, for example, the image transmission and storage process corresponding to the image capture process 600A is the image transmission and storage process 610A.

[0091] In the example shown in Figure 11, the image capture process and the distance measurement process are repeatedly executed in parallel. In a pair of parallel-executed image capture and distance measurement processes, a pair of images 240 and acquired distances 303 are obtained. For example, in a pair of parallel-executed image capture processes 600A and distance measurement processes 630A, a pair of images 240A and acquired distances 303A are obtained. Also in the example shown in Figure 11, the image capture process, image transmission and storage process, 2D position determination process, and 3D position determination process are pipelined. Furthermore, the distance measurement process, distance transmission and storage process, and 3D position determination process are pipelined. As a result, the processing system 1 can execute each of the image capture process, image transmission and storage process, 2D position determination process, distance measurement process, distance transmission and storage process, and 3D position determination process in an assembly-line manner. As a result, the interval for determining the 3D position of the processing device 3 can be shortened. In the example shown in Figure 11, it can be said that the position determination process is repeatedly executed in parallel.

[0092] For example, processing system 1 can execute the image capture process 600B immediately after the image capture process 600A, regardless of whether the image transmission and storage process 610A, the two-dimensional positioning process 620A, and the three-dimensional positioning process 650A are executed. Also, processing system 1 can execute the image transmission and storage process 610B immediately after the image capture process 600B, regardless of whether the two-dimensional positioning process 620A and the three-dimensional positioning process 650A are executed. Also, processing system 1 can execute the two-dimensional positioning process 620B immediately after the image transmission and storage process 610B, regardless of whether the three-dimensional positioning process 650A is executed. Also, processing system 1 can execute the distance measurement process 630B immediately after the distance measurement process 630A, regardless of whether the distance transmission and storage process 640A and the three-dimensional positioning process 650A are executed. Also, processing system 1 can execute the distance transmission and storage process 640B immediately after the distance measurement process 630B, regardless of whether the three-dimensional positioning process 650A is executed.

[0093] In the example shown in Figure 11, in the frame period 250 following the frame period 250 in which the imaging process is performed, a 3D positioning process corresponding to the imaging process is executed to determine the 3D position of the processing unit 3. The 3D position of the processing unit 3 is then repeatedly determined at intervals equal to the length of the frame period 250, in other words, at the same interval as the execution interval of the imaging process.

[0094] In the example shown in Figure 11, the completion timing of the image transmission and storage process is earlier than the completion timing of the corresponding distance transmission and storage process. On the other hand, if the communication speed between the camera 24 and the control unit 20 is low due to the modularization of the camera 24 or other reasons, the completion timing of the image transmission and storage process may be later than the completion timing of the corresponding distance transmission and storage process. Figure 12 is a schematic diagram showing an example of the operation of the processing system 1 in this case.

[0095] Figure 12 shows six consecutive frame periods 250A, 250B, 250C, 250D, 250E, and 250F. In the example in Figure 12, the end timing of the image transmission and storage process is delayed by approximately the length of two frame periods 250 compared to the end timing of the corresponding distance transmission and storage process. As a result, in frame period 250 three frames after the frame period 250 in which the image capture process takes place, the 3D position determination process corresponding to the image capture process is executed, and the 3D position of the processing unit 3 is determined. Meanwhile, the 3D position of the processing unit 3 is determined repeatedly at intervals equal to the length of the frame periods 250, similar to the example in Figure 11.

[0096] As shown in the examples in Figures 11 and 12, when pipeline processing is performed in processing system 1, the control unit 20 of processing unit 2 instructs camera 24 to start shooting, and after the first shooting process 600A is executed, it associates the nth image 240 (where n is a variable, an integer satisfying 1 ≤ n) obtained with the nth acquired distance 303 sent from processing unit 3. Then, the control unit 20 uses the associated image 240 and acquired distance 303 to determine the 3D position of processing unit 3. In other words, in the 3D position determination process, the control unit 20 determines the 3D position of processing unit 3 based on the acquired distance 303 associated with the image 240 used in the corresponding 2D position determination process. As a result, the image 240 and acquired distance 303 obtained in the same frame period 250 are used to appropriately determine the 3D position of processing unit 3. In other words, the pair of images 240 and acquired distance 303 obtained in a pair of parallel-executed shooting and distance measurement processes are used to appropriately determine the 3D position of processing unit 3. Hereafter, when we simply refer to the paired images 240 and acquired distance 303, we mean the paired images 240 and acquired distance 303 obtained by a pair of parallel-executed imaging and distance measurement processes.

[0097] Here, even though the processing unit 2 is emitting light 220 and sound waves 230, it may not be able to receive the acquired distance 303 from the processing unit 3. For example, if a person is in front of the light receiving unit 32 and the light receiving unit 32 cannot receive light 220, the distance acquisition unit 302 cannot determine the distance between the processing unit 3 and the processing unit 2. In this case, although the processing unit 2 is emitting light 220 and sound waves 230, it cannot receive the acquired distance 303 from the processing unit 3.

[0098] If the processing unit 2 is unable to receive the acquired distance 303, and the image 240 and the acquired distance 303 are associated in the manner described above, the corresponding image 240 and acquired distance 303 may not be used to determine the 3D position of the processing unit 3. For example, consider the case where the acquired distance 303B is not transmitted from the processing unit 3. In this case, the second image 240 obtained after the execution of the imaging process 600A is image 240B, but the second acquired distance 303 sent from the processing unit 3 after the execution of the imaging process 600A is acquired distance 303C, and acquired distance 303C is associated with image 240B. Therefore, the 3D position is determined based on image 240B and the acquired distance 303C obtained in a different frame period 250 than the frame period 250 in which it was obtained. As a result, the accuracy of the determined 3D position 209 may decrease.

[0099] The following describes a processing system 1 that enables the appropriate identification of the paired image 240 and acquisition distance 303.

[0100] Figure 13 is a schematic diagram showing another example of the control unit 20 of the processing unit 2. In the example of Figure 13, the control unit 20 further includes a identification unit 215. The identification unit 215 is capable of identifying a pair of images 240 and acquisition distances 303 based on predetermined information. In the two-dimensional positioning process and the three-dimensional positioning process, the pair of images 240 and acquisition distances 303 identified by the identification unit 215 are used, respectively.

[0101] Here, with respect to the paired image 240 and acquisition distance 303, the timing at which the image 240 becomes available to the 2D positioning unit 206 is defined as the first timing, and the timing at which the acquisition distance 303 becomes available to the 3D positioning unit 208 is defined as the second timing. In the processing system 1, for example, when the image 240 is stored in the storage unit 21, the image 240 becomes available to the 2D positioning unit 206. The first timing is, for example, the timing at which the image 240 is stored in the storage unit 21. In other words, the first timing is, for example, the timing at which the image transmission and storage processing is completed. Also, in the processing system 1, for example, when the acquisition distance 303 is stored in the storage unit 21, the acquisition distance 303 becomes available to the 3D positioning unit 208. The second timing is, for example, the timing at which the acquisition distance 303 is stored in the storage unit 21. In other words, the second timing is, for example, the timing at which the distance transmission and storage processing is completed.

[0102] The identification unit 215 identifies a pair of images 240 and acquisition distance 303 based on the difference between the first timing and the second timing (also called the usable timing difference). The storage unit 21 stores the previously determined usable timing difference. The identification unit 215 identifies a pair of images 240 and acquisition distance 303 based on the usable timing difference stored in the storage unit 21.

[0103] Figures 11 and 12 show an example of a usable timing difference of 500. In the example in Figure 11, the first timing (i.e., the end timing of the image transmission and storage process) is earlier than the second timing (i.e., the end timing of the distance transmission and storage process). On the other hand, in the example in Figure 12, the first timing is later than the second timing. The usable timing difference can be determined in advance, for example, using actual equipment or simulation.

[0104] The available timing difference can be expressed as a positive or negative value. For example, if the available timing difference is a positive value, the first timing is earlier than the second timing, and if the available timing difference is a negative value, the first timing is later than the second timing.

[0105] If the first timing is earlier than the second timing, as in the example in Figure 11, the identification unit 215, for example, when the image 240 becomes available in the two-dimensional position identification unit 206, designates the image 240 as the image to be processed 240. In other words, the identification unit 215 designates the image 240 as the image to be processed 240 when the image transmission and storage processing is completed and the image 240 is stored in the storage unit 21. The identification unit 215 then determines whether there is an acquisition distance 303 (also called the acquisition distance 303 to be associated) that has become available for use in the three-dimensional position identification unit 208, near a timing that is delayed by the absolute value of the difference in available timing from the timing when the image to be processed 240 becomes available in the two-dimensional position identification unit 206. In other words, the identification unit 215 determines whether there is an acquisition distance 303 stored in the storage unit 21, near a timing that is delayed by the absolute value of the difference in available timing from the completion timing of the image transmission and storage processing of the image to be processed 240. The identification unit 215 associates the acquisition distance 303 to be associated with the image 240 to be processed if such an acquisition distance 303 exists. On the other hand, if the acquisition distance 303 to be associated with the image 240 does not exist, the identification unit 215 does not associate the acquisition distance 303 with the image 240 to be processed, for example, by discarding the image 240. The identification unit 215 then considers the image 240 and acquisition distance 303 that have been associated with each other as a pair of images 240 and acquisition distance 303. This ensures that even if the processing device 3 does not transmit the acquisition distance 303, the pair of images 240 and acquisition distance 303 can be appropriately identified.

[0106] For example, consider the case in the example in Figure 11 where the image to be processed 240 is image 240B, and the acquisition distance 303B is not transmitted from the processing unit 3. In this case, there is no acquisition distance 303 to be associated with that is stored in the storage unit 21 at a timing that is later by the absolute value of the difference in available timings from the end timing of the image transmission and storage processing of the image to be processed 240B. Therefore, the acquisition distance 303 is not associated with the image to be processed 240B, and the image to be processed 240B is discarded. Thus, the 3D position is not determined based on the image to be processed 240B and the acquisition distance 303 which is not paired with it.

[0107] Furthermore, as shown in the example in Figure 12, if the second timing is earlier than the first timing, the identification unit 215 sets the acquired distance 303 as the target acquired distance 303 for processing when the acquired distance 303 becomes available in the 3D position identification unit 208. In other words, the identification unit 215 sets the acquired distance 303 as the target acquired distance 303 for processing when the distance transmission storage processing is completed and the acquired distance 303 is stored in the storage unit 21. The identification unit 215 then determines whether or not there is an image 240 (the image to be associated 240) that has become available in the 2D position identification unit 206, near a timing that is delayed by the absolute value of the difference in available timings from the timing when the target acquired distance 303 becomes available in the 3D position identification unit 208. In other words, the identification unit 215 determines whether or not there is an image 240 stored in the storage unit 21, near a timing that is delayed by the absolute value of the difference in available timings from the completion timing of the distance transmission storage processing of the target acquired distance 303. The identification unit 215 associates the image 240 to be associated with the acquisition distance 303 to be processed if such an image 240 exists. On the other hand, if the image 240 to be associated with the acquisition distance 303 does not exist, the identification unit 215 associates the image 240 with the acquisition distance 303 to be processed, for example, by discarding the acquisition distance 303. The identification unit 215 then considers the associated image 240 and acquisition distance 303 as a pair. This allows for the appropriate identification of the paired image 240 and acquisition distance 303.

[0108] If the distance acquisition unit 302 is provided in the processing unit 2 instead of the processing unit 3, the timing at which the acquired distance 303 becomes available to the 3D position identification unit 208 of the processing unit 2 (also called the distance availability timing) is, for example, the timing at which the acquired distance 303 determined by the distance acquisition unit 302 is stored in the storage unit 21. Also, if the 2D position identification unit 206 is provided in the processing unit 3 instead of the processing unit 2, the timing at which the image 240 becomes available to the 2D position identification unit 206 (also called the image availability timing) is, for example, the timing at which the image 240 is stored in the storage unit 31 of the processing unit 3. Furthermore, if the 3D position identification unit 208 is provided in the processing unit 3 instead of the processing unit 2, the distance availability timing is, for example, the timing at which the acquired distance 303 determined by the distance acquisition unit 302 is stored in the storage unit 31.

[0109] The identification unit 215 may be provided in the control unit 30 of the processing device 3 instead of the control unit 20. Regardless of where the identification unit 215, distance acquisition unit 302, two-dimensional position identification unit 206, and three-dimensional position identification unit 208 are provided, the identification unit 215 can identify a pair of images 240 and acquired distances 303 based on a predetermined difference in available timings, provided that it knows the timing when images can be used and when distances can be used.

[0110] For example, consider a case where the identification unit 215 and the distance acquisition unit 302 are provided in the processing unit 3, and the two-dimensional position identification unit 206 and the three-dimensional position identification unit 208 are provided in the processing unit 2. In this case, the control unit 20 of the processing unit 2 identifies the image availability timing for each image 240 generated by the camera 24 and the distance availability timing for each acquired distance 303 transmitted from the processing unit 3, and notifies the processing unit 3. As a result, the identification unit 215 of the processing unit 3 can know the image availability timing for each image 240 and the distance availability timing for each acquired distance 303, and can identify a pair of images 240 and acquired distances 303 in the same manner as described above.

[0111] Another example is when the distance acquisition unit 302 and the 3D position identification unit 208 are provided in the processing unit 3, and the identification unit 215 and the 2D position identification unit 206 are provided in the processing unit 2. In this case, the control unit 30 of the processing unit 3 identifies the distance availability timing for each acquired distance 303 obtained by the distance acquisition unit 302 and notifies the processing unit 2. On the other hand, the control unit 20 of the processing unit 2 identifies the image availability timing for each image 240 generated by the camera 24. As a result, the identification unit 215 of the processing unit 2 can know the image availability timing for each image 240 and the distance availability timing for each acquired distance 303, and can identify the pair of images 240 and acquired distances 303 in the same manner as described above.

[0112] Another example is when the distance acquisition unit 302 and the two-dimensional position identification unit 206 are provided in the processing unit 3, and the identification unit 215 and the three-dimensional position identification unit 208 are provided in the processing unit 2. In this case, the control unit 30 of the processing unit 3 identifies the image availability timing for each image 240 transmitted from the processing unit 2 and notifies the processing unit 2. On the other hand, the control unit 20 of the processing unit 2 identifies the distance availability timing for each acquired distance 303 transmitted from the processing unit 3. As a result, the identification unit 215 of the processing unit 2 can know the image availability timing for each image 240 and the distance availability timing for each acquired distance 303, and can identify the pair of images 240 and acquired distances 303 in the same manner as described above.

[0113] Processing system 1 may include multiple processing units 3, as shown in the example in Figure 14. In this case, the processing system 1 may specify the three-dimensional position of each of the multiple processing units 3. In the example in Figure 14, the multiple processing units 3 included in processing system 1 consist of, for example, four processing units 3a, 3b, 3c, and 3d. The number of multiple processing units 3 included in processing system 1 is not limited to this.

[0114] In the example shown in Figure 14, the processing unit 2 performs step s1 described above, and each processing unit 3 performs steps s11 and s12 in response to the execution of step s1. As a result, each of the multiple processing units 3 determines the distance between itself and the processing unit 2 through one emission and transmission of light in the processing unit 2. The acquired distance 303 determined by each processing unit 3 is transmitted to the processing unit 2. In the processing unit 2, the camera 24 captures, for example, the displays 340 of the multiple processing units 3. The control unit 20 then identifies the two-dimensional position of the display 340 of each processing unit 3 in the image 240 obtained by the camera 24. In other words, the control unit 20 includes multiple two-dimensional position identification units 206 that identify the two-dimensional position of each of the displays 340 of the multiple processing units 3. The control unit 20 then identifies the three-dimensional position of each processing unit 3 based on the acquired distance 303 from the processing unit 3 and the identified two-dimensional position 207 for the processing unit 3. In other words, the control unit 20 includes multiple three-dimensional position identification units 208 that identify the three-dimensional position of each of the multiple processing units 3. Between the processing unit 2 and each processing unit 3, the process shown in Figure 8 may be executed, or the processes shown in Figures 11 and 12 may be executed.

[0115] Some of the multiple processing units 3 may have different configurations from each other. For example, some of the multiple processing units 3 may have an operating unit 35, while the remaining parts of the multiple processing units 3 may not have an operating unit 35. Also, some of the multiple processing units 3 may have an inertial sensor 36, while the remaining parts of the multiple processing units 3 may not have an inertial sensor 36.

[0116] Furthermore, the distance acquisition unit 302 of at least one of the multiple processing units 3 may be provided in the processing unit 2. In other words, the processing unit 2 may have at least one distance acquisition unit 302 that determines the distance between the processing unit 3 and the processing unit 2. Also, at least one of the multiple two-dimensional position identification units 206 provided in the processing unit 2 may be provided in the processing unit 3 in which the at least one two-dimensional position identification unit 206 identifies the two-dimensional position of its display 340. In other words, the multiple processing units 3 may include at least one processing unit 3 having a two-dimensional position identification unit 206 that identifies the two-dimensional position of its own display 340. Also, at least one of the multiple three-dimensional position identification units 208 provided in the processing unit 2 may be provided in the processing unit 3 in which the at least one three-dimensional position identification unit 208 identifies its three-dimensional position. In other words, the multiple processing units 3 may include at least one processing unit 3 having a three-dimensional position identification unit 208 that identifies its own three-dimensional position.

[0117] The displays 340 of multiple processing units 3 may be the same or different. Figure 15 is a schematic diagram showing an example of how processing units 3a, 3b, 3c, and 3d display different displays 340. In Figure 15, the displays 340 of processing units 3a, 3b, 3c, and 3d are shown as displays 340a, 340b, 340c, and 340d, respectively. In Figure 15, each display 340 is surrounded by a dashed line to distinguish it from others, but this dashed line is not included in the display 340.

[0118] In the example in Figure 15, each of the displays 340a, 340b, 340c, and 340d is a dot pattern consisting of three dots. The arrangement of the three dots differs among the displays 340a, 340b, 340c, and 340d.

[0119] Thus, when the displays 340 of multiple processing units 3 are different from each other, the control unit 20 of the processing unit 2 can use pattern recognition processing or the like to identify which processing unit 3 each display 340 in the image 240 belongs to. This allows the control unit 20 to appropriately identify which processing unit 3's 3D position each specific 3D position 209 identified based on the 2D position of the display 340 belongs to. Therefore, the control unit 20 can independently manage the 3D position of each processing unit 3. For example, the control unit 20 may control the display of the display device 10 for each processing unit 3 based on its 3D position. It should be noted that only some of the displays 340 of the multiple processing units 3 may be different from each other.

[0120] The functions of the elements disclosed herein may be implemented using circuit configurations or processing circuit configurations, including general-purpose processors, dedicated processors, integrated circuits, ASICs ("Application-Specific Integrated Circuits"), conventional circuit configurations, and / or combinations thereof, configured to perform the disclosed elements or programmed to perform the disclosed functions. A processor is considered a processing circuit configuration or circuit configuration if it includes transistors and other circuit configurations within it. In this disclosure, a circuit configuration, unit, or means is hardware that performs the listed functions or hardware programmed to perform such functions. Hardware may be any hardware disclosed herein or other known hardware programmed to perform the listed functions or configured to perform such functions. When hardware is a processor that may be considered a type of circuit configuration, a circuit configuration, means, or unit is a combination of hardware and software, software used to configure the hardware, and / or a processor.

[0121] As described above, the processing system and processing apparatus have been explained in detail, but the above description is illustrative in all respects, and the invention is not limited thereto. Furthermore, the various modifications described above can be applied in combination as long as they do not contradict each other. And it is understood that countless modifications not illustrated can be envisioned without falling outside the scope of the invention. [Explanation of Symbols]

[0122] 1. Processing System 2,3,3a,3b,3c,3d processing equipment 22 Light-emitting part 23 Transmitter 24 cameras 32 Light receiving part 33 Receiving section 34 Display section 206 2D position identification part 208 3D position identification part 220 light 215 Specific part 230 sound waves 302 Distance acquisition part 306 Sensitivity adjustment section 340,340a,340b,340c,340d display

Claims

1. A first processing device having a light-emitting unit that emits light, a sound-emitting unit that emits sound waves, and a camera, A second processing apparatus having a first light receiving unit for receiving light, a first wave receiving unit for receiving sound waves, and a first display unit for performing a first display. Equipped with, The camera performs a shooting process to capture the first display, A first distance acquisition unit performs distance acquisition processing to determine a first distance between the second processing unit and the first processing unit based on the difference between the first light reception timing of the light at the first light receiving unit and the first sound wave reception timing of the sound wave at the first wave receiving unit. A first two-dimensional position identification unit performs a two-dimensional position identification process to identify the first two-dimensional position of the first display in the image obtained by the aforementioned shooting process, A first three-dimensional positioning unit performs a three-dimensional positioning process to determine the first three-dimensional position of the second processing unit based on the first two-dimensional position and the first distance. Equipped with, A processing system in which the distance measurement process and the imaging process are executed in parallel such that at least a portion of the execution period of the distance measurement process, which includes the emission of light from the light-emitting unit, the transmission of waves from the wave-transmitting unit, the reception of light from the first light-receiving unit, the reception of waves from the first wave-receiving unit, and the distance acquisition process, overlaps with at least a portion of the execution period of the imaging process.

2. A processing system according to claim 1, A processing system in which the first three-dimensional position is repeatedly determined by pipeline processing of the aforementioned imaging process, the two-dimensional position determination process, and the three-dimensional position determination process, as well as by pipeline processing of the aforementioned distance measurement process and the three-dimensional position determination process.

3. The processing system according to claim 2, With respect to a pair of images and a first distance obtained in a pair of imaging and distance measurement processes executed in parallel, the timing at which the images become available to the first two-dimensional positioning unit is defined as the first timing, and the timing at which the first distance becomes available to the first three-dimensional positioning unit is defined as the second timing. The system includes a specification unit that specifies the pair of images and the first distance based on the difference between the first timing and the second timing. A processing system in which the pair of images and the first distance identified by the identification unit are used in the two-dimensional position identification process and the three-dimensional position identification process, respectively.

4. A first processing apparatus having a light-emitting unit that emits light, a sound-emitting unit that emits sound waves, and a camera, A second processing apparatus having a first light receiving unit for receiving light, a first wave receiving unit for receiving sound waves, and a first display unit for performing a first display. Equipped with, The camera performs a shooting process to capture the first display, A first distance acquisition unit performs distance acquisition processing to determine a first distance between the second processing unit and the first processing unit based on the difference between the first light reception timing of the light at the first light receiving unit and the first sound wave reception timing of the sound wave at the first wave receiving unit. A first two-dimensional position identification unit performs a two-dimensional position identification process to identify the first two-dimensional position of the first display in the image obtained by the aforementioned shooting process, A first three-dimensional positioning unit performs a three-dimensional positioning process to determine the first three-dimensional position of the second processing unit based on the first two-dimensional position and the first distance. Equipped with, The second processing device is a processing system comprising a sensitivity adjustment unit that increases the reception sensitivity of the first wave receiving unit in proportion to the elapsed time from the first light reception timing to the first wave receiving timing.

5. A processing system according to any one of claims 1 to 4, The second processing device is a processing system having the first distance acquisition unit.

6. A processing system according to any one of claims 1 to 5, The first processing device is a processing system having the first two-dimensional positioning unit.

7. A processing system according to any one of claims 1 to 6, The first processing device performs processing based on the first three-dimensional position, The first processing device is a processing system having the first three-dimensional positioning unit.

8. A processing system according to any one of claims 1 to 7, The aforementioned light is infrared radiation, in this processing system.

9. A processing system according to any one of claims 1 to 8, The aforementioned sound waves are ultrasonic waves in the processing system.

10. A processing system according to any one of claims 1 to 9, The processing system wherein the first display unit is an infrared light emitting unit.

11. A processing system according to any one of claims 1 to 10, The third processing apparatus includes a second light receiving unit for receiving the aforementioned light, a second wave receiving unit for receiving the aforementioned sound waves, and a second display unit for performing a second display. The camera captures the first display and the second display, A second distance acquisition unit that determines a second distance between the third processing unit and the first processing unit based on the difference between the second light reception timing of the light at the second light receiving unit and the second sound wave reception timing of the sound wave at the second wave receiving unit, A second two-dimensional position identification unit for identifying the second two-dimensional position of the second display in the aforementioned image, A second three-dimensional positioning unit that determines the second three-dimensional position of the third processing device based on the second two-dimensional position and the second distance. A processing system equipped with the following features.

12. The processing system according to claim 11, A processing system in which the first display and the second display are different displays from each other.

13. A first processing apparatus comprising the processing system according to any one of claims 1 to 12.

14. A second processing apparatus comprising the processing system according to any one of claims 1 to 12.

Images