Information processing device, information processing system, information processing method, and information processing program

Abstract

The present invention provides an information processing device, an information processing system, an information processing method, and an information processing program that can improve the accuracy of detecting the user's hand movements. [Solution] An information processing device according to one aspect of the present disclosure is an information processing device that operates based on the movement of a user's hand, and comprises a first image acquisition unit, an image synthesis unit, a second image acquisition unit, and a detection unit. The first image acquisition unit acquires a first image including the user's body captured by a first camera and a second image including the user's wrist captured by a second camera. The image synthesis unit uses the first and second images to synthesize a third image including the body and wrist. The second image acquisition unit identifies a plurality of first feature points corresponding to the body from the third image and acquires a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. The detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image and detects hand movement based on the plurality of second feature points.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, in various devices, there are those that detect the movement of a user's hand and perform operation control according to the detected hand movement (see, for example, Patent Document 1). For example, for an operation target such as a display that a user's hand cannot reach or an operation device in a vehicle interior where a user points to an object outside the vehicle, there is a device that changes the content displayed on the operation device according to the movement of the user's hand without the user directly touching the operation device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described technology, for example, as a process of detecting the movement of a user's hand, an image captured by a camera is acquired, the user's body and hand are identified from the acquired image, and the movement of the user's hand is detected based on the direction pointed by the identified hand. However, a camera mounted in a vehicle interior may not be able to capture both the user's body and hand within the camera's angle of view, and the acquired image may not include the user's body and hand. Therefore, in the process of detecting the movement of the user's hand, the user's body and hand may not be identified from the acquired image, and the movement of the user's hand may not be detected.

[0005] The present disclosure provides an information processing apparatus, an information processing system, an information processing method, and an information processing program capable of improving the accuracy of detecting the movement of a user's hand. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure is an information processing device that operates based on the movement of a user's hand, and comprises a first image acquisition unit, an image synthesis unit, a second image acquisition unit, and a detection unit. The first image acquisition unit acquires a first image including the user's body captured by a first camera and a second image including the user's wrist captured by a second camera. The image synthesis unit uses the first and second images to synthesize a third image including the body and wrist. The second image acquisition unit identifies a plurality of first feature points corresponding to the body from the third image and acquires a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. The detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image and detects hand movement based on the plurality of second feature points. [Effects of the Invention]

[0007] The information processing device described herein can improve the accuracy of detecting the user's hand movements. The effects described herein are not necessarily limited to those described herein and may include any of the effects described herein. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing an example of an information processing device. [Figure 2] Figure 2 is a schematic diagram illustrating the processing performed by the information processing device. [Figure 3] Figure 3 is a schematic diagram illustrating the arrangement of sensors in the first comparative example. [Figure 4] Figure 4 is a schematic diagram illustrating the arrangement of sensors according to the first embodiment. [Figure 5] Figure 5 is a diagram showing the schematic configuration of an information system including a control device, which is an information processing device according to the first embodiment. [Figure 6] Figure 6 is a schematic diagram illustrating the processing of the control device according to the first embodiment. [Figure 7]Figure 7 is a schematic diagram illustrating the processing of the control device according to the first embodiment. [Figure 8] Figure 8 is a schematic diagram illustrating the processing of the control device according to the first embodiment. [Figure 9] Figure 9 is a schematic diagram illustrating the processing of the control device according to the first embodiment. [Figure 10] Figure 10 is a flowchart showing the processing procedure of the control device according to the first embodiment. [Figure 11] Figure 11 is a diagram showing the schematic configuration of an information system including a control device, which is an information processing device according to the second embodiment. [Figure 12] Figure 12 is a schematic diagram illustrating the processing of the control device according to the second embodiment. [Figure 13] Figure 13 is a flowchart showing the processing procedure of the control device according to the second embodiment. [Figure 14] Figure 14 is a schematic diagram illustrating the processing of the control device according to the first modified example. [Figure 15] Figure 15 is a schematic diagram illustrating the processing of the control device according to the second modified example. [Figure 16] Figure 16 is a schematic diagram illustrating the processing of the control device according to the third modified example. [Figure 17] Figure 17 is a schematic diagram illustrating the processing of the control device according to the third embodiment. [Figure 18] Figure 18 is a diagram showing the schematic configuration of an information system including a control device, which is an information processing device according to the fourth embodiment. [Figure 19] Figure 19 is a schematic diagram illustrating the processing of the control device according to the fourth embodiment. [Figure 20] Figure 20 is a flowchart showing the processing procedure of the control device according to the fourth embodiment. [Figure 21] Figure 21 is a schematic diagram illustrating the arrangement of sensors according to the fifth embodiment. [Figure 22] Figure 22 is a schematic diagram illustrating the arrangement of sensors according to the fifth embodiment. [Figure 23] FIG. 23 is a schematic diagram for explaining the content of the processing of the control device 1 according to the fifth embodiment. [Figure 24] FIG. 24 is a schematic diagram for explaining the arrangement of sensors according to the fifth modification. [Figure 25] FIG. 25 is a schematic diagram for explaining the content of the processing of the control device according to the fifth modification. [Figure 26] FIG. 26 is a schematic diagram for explaining the arrangement of sensors according to the sixth modification. [Figure 27] FIG. 27 is a schematic diagram for explaining the content of the processing of the control device according to the sixth modification. [Figure 28] FIG. 28 is a diagram showing an overview configuration of an information system including a control device which is an information processing device according to the sixth embodiment. [Figure 29] FIG. 29 is a schematic diagram for explaining the content of the processing of the control device according to the sixth embodiment. [Figure 30] FIG. 30 is a flowchart showing the processing procedure of the control device according to the sixth embodiment.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the information processing device according to the present disclosure will be described with reference to the drawings.

[0010] (First Embodiment) Before describing the information processing device according to the first embodiment, a device that operates based on the movement of a user's hand will be described.

[0011] Figure 1 is a schematic diagram showing an example of an information processing device. Figure 1 shows an example of an information processing device that operates based on the user's hand movements. The information processing device has a sensor 500 and an operating device 600. The sensor 500 is, for example, a camera. The sensor 500 captures an image of the user who will be operating the device. The user has an arm, which includes not only the upper arm and forearm but also the hand. The hand also includes the fingertips 72. The operating device 600 is, for example, a display device such as a display mounted in a vehicle. The information processing device controls the data displayed by the operating device 600 based on the position of the user's fingertips 72.

[0012] Furthermore, for example, if a user points to a building outside the vehicle from inside the vehicle, the information processing device controls the operating device 600 to display the direction in which the user's fingertip 72 is pointing, based on the position of the user's fingertip 72. The object pointed to by the user's fingertip 72 includes the operating device 600 and objects such as buildings outside the vehicle.

[0013] Furthermore, the information processing device calculates the angle required to receive an operation signal to the device. Specifically, the information processing device calculates an angle 900 which represents the angle between the vehicle's direction of travel M1 and the angle line 700 that points to the operating device 600 in the XZ plane with the user's fingertip 72.

[0014] In Figure 1 and the drawings relating to the operating device 600 described below, the mutually orthogonal X, Y, and Z axes represent the left-right, up-down, and front-back directions of the operating device 600, respectively. Furthermore, in the following description, when simply referred to as the X direction, Y direction, or Z direction, it means the respective axis direction, including the two opposite directions.

[0015] Furthermore, if the positive direction of the X-axis is specified, it is a one-way direction from left to right; if the positive direction of the Y-axis is specified, it is a one-way direction from bottom to top; and if the positive direction of the Z-axis is specified, it is a one-way direction from front to back. If the negative direction of the X-axis is specified, it is a one-way direction from right to left; if the negative direction of the Y-axis is specified, it is a one-way direction from top to bottom; and if the negative direction of the Z-axis is specified, it is a one-way direction from back to front.

[0016] Figure 2 is a schematic diagram illustrating the processing of the information processing device. Figure 2 illustrates the process by which the information processing device detects the user's fingertip 72. First, the information processing device acquires an image 201 of the user captured by the sensor 500. The information processing device also identifies multiple feature points 203 corresponding to the user's body and wrist from the acquired image 202.

[0017] Furthermore, the information processing device acquires an image 204 containing multiple feature points 203 corresponding to the user's wrist. The information processing device then identifies multiple feature points 205 corresponding to the user's hand from the acquired image 204 and detects the user's fingertips 72. Upon detecting the user's fingertips 72, the information processing device controls the data displayed by the operating device 600 based on the position of the user's fingertips 72.

[0018] Figure 3 is a schematic diagram illustrating the arrangement of the sensor 500 in the first comparative example. The schematic diagram in Figure 3 shows the arrangement state D1 when the sensor 500 is mounted in a virtual position capable of capturing the image 201 shown in Figure 2. Here, the user's hand is assumed to be located on the steering wheel 721 in the vehicle.

[0019] For example, in order to capture image 201 shown in Figure 2, the sensor 500 may be positioned outside the vehicle, as in arrangement state D1. If the sensor 500 were placed inside the vehicle, the field of view of the sensor 500 would not match, and therefore the user's body and wrist would not be included in the image captured by the sensor 500.

[0020] Therefore, the information processing device may not be able to identify the user's wrist and thus may not be able to detect the user's fingertips 72. This embodiment provides an information processing device that can improve the accuracy of detecting the user's hand movements by, for example, mounting the sensor 500 in a position where it can capture an image including the user's body and wrist inside the vehicle.

[0021] Figure 4 is a schematic diagram illustrating the arrangement of the sensor 500 according to the first embodiment. The schematic diagram shown in Figure 4 shows the arrangement state D2 when the sensor 500 is mounted in a position inside the vehicle interior where the image 201 shown in Figure 2 can be captured.

[0022] For example, as shown in arrangement state D2, the sensor 500 has a first sensor 501 and a second sensor 502. The first sensor 501 and the second sensor 502 are positioned in front of and in front of the user inside the vehicle cabin. For example, the first sensor 501 captures an image including the user's body and is positioned near a rear-view mirror mounted inside the vehicle cabin.

[0023] Furthermore, for example, the second sensor 502 captures an image including the user's wrist and is positioned directly above the seat where the user sits in the vehicle compartment. By arranging the first sensor 501 and the second sensor 502 as described above, it is possible to capture an image including the user's body and hands in the vehicle compartment.

[0024] Figure 5 is a diagram showing the schematic configuration of an information system including a control device, which is an information processing device according to the first embodiment. The information processing device according to the first embodiment comprises a sensor 500, an operating device 600, and a control device 1.

[0025] Sensor 500 is, for example, a camera device. The camera device is, as an example, an infrared camera. However, the camera device is not limited to an infrared camera and includes visible light cameras. Sensor 500 is an example of an imaging unit. Sensor 500 captures an image of the user, who is the operator, and outputs the captured image to the control device 1.

[0026] Sensor 500 includes a first sensor 501 and a second sensor 502. The first sensor 501 is also called the first camera. The second sensor 502 is also called the second camera. For example, the first sensor 501 captures an image including the user's body. Also, for example, the second sensor 502 captures an image including the user's wrist. Sensor 500 continuously performs imaging processing and outputs the image to the control device 1.

[0027] Sensor 500 is mounted inside the vehicle's cabin. Specifically, the first sensor 501 and the second sensor 502 are positioned spaced apart as shown in Figure 4, and the optical axes of the first sensor 501 and the second sensor 502 intersect. For example, the first sensor 501 is positioned directly in front of the user and in the positive Y-axis direction and the negative Z-axis direction on the YZ plane. In other words, the first sensor 501 is positioned diagonally above the user.

[0028] Furthermore, for example, the second sensor 502 is positioned in the positive direction of both the Y-axis and the Z-axis on the YZ plane, relative to the user. In other words, the second sensor 502 is positioned diagonally behind the user.

[0029] The operating device 600 is a display unit that displays various data. The operating device 600 is, for example, a display device such as a display mounted on a vehicle. The control device 1 processes the data displayed on the operating device 600 in response to the movement of the user's fingertip 72 detected via the sensor 500.

[0030] The control device 1 comprises a control unit 10 and a storage unit 30. The control unit 10 is configured, for example, as a CPU (Central Processing Unit) and comprehensively controls the operation of each part of the control device 1. The control device 1 according to this embodiment has a ROM and RAM (not shown). The ROM stores various programs. The RAM is a work area when the CPU executes programs.

[0031] The control device 1 includes, for example, a processor and memory, and the processor executes a program stored in memory to realize the functions of the control unit 10 and each functional block of the control unit 10. A CPU is an example of a processor. A storage unit 30 is an example of memory.

[0032] The CPU uses RAM as a working area and executes a program stored in ROM to realize the first image acquisition unit 11, image conversion unit 12, image synthesis unit 13, second image acquisition unit 14, and detection unit 15, as shown in Figure 5. This can also be rephrased as the control device 1 comprising the first image acquisition unit 11, image conversion unit 12, image synthesis unit 13, second image acquisition unit 14, and detection unit 15. The first image acquisition unit 11, image conversion unit 12, image synthesis unit 13, second image acquisition unit 14, and detection unit 15 may be realized by different hardware.

[0033] The first image acquisition unit 11 acquires a first image including the user's body captured by the first sensor 501, and a second image including the user's wrist captured by the second sensor 502. The first and second images are captured so that the optical axes of the first sensor 501 and the second sensor 502 intersect.

[0034] The image conversion unit 12 converts the first image into a projected image that corresponds to the coordinate system of the second camera. Specifically, the image conversion unit 12 converts the first image acquired by the first image acquisition unit 11 into a projected image that corresponds to the coordinate system of the second camera.

[0035] Here, the first image and the projected image will be explained using Figure 6. Figure 6 is a schematic diagram for explaining the processing of the control device 1 according to the first embodiment. In Figure 6, the processing performed by the image conversion unit 12 of the control unit 10 of the control device 1 will be explained. Figure 6 shows the first image 211, which includes the user's body and was acquired by the first image acquisition unit 11, and the projected image 216, which was obtained by the image conversion unit 12 by projecting the first image 211.

[0036] The first image 211 shows coordinates 212, 213, 214, and 215 in the first sensor 501. The image transformation unit 12 performs a projection transformation on the first image 211 to correspond to the coordinate system in the second sensor 502, for example, projecting from coordinate 212 to coordinate 217. Similarly, the image transformation unit 12 performs projection transformations from coordinate 213 to coordinate 218, from coordinate 214 to coordinate 219, and from coordinate 215 to coordinate 220.

[0037] Returning to Figure 5, let's continue the explanation. The image synthesis unit 13 uses the first image and the second image to synthesize them into a third image that includes the body and wrists. Specifically, the image synthesis unit 13 uses the first image and the second image acquired by the first image acquisition unit 11 to synthesize them into a third image that includes the user's body and wrists. For example, the image synthesis unit 13 uses the projected image 216 obtained by the image conversion unit 12 projecting the first image 211, and the optical axis converted image obtained by optically converting the second image so that it aligns with the optical axis of the first sensor 501, to synthesize them into a third image that includes the user's body and wrists.

[0038] Here, a third image including the user's body and wrist will be explained using Figures 7 and 8. Figures 7 and 8 are schematic diagrams illustrating the processing of the control device 1 according to the first embodiment. In Figures 7 and 8, the processing performed by the image synthesis unit 13 of the control unit 10 of the control device 1 will be explained.

[0039] Figure 7 shows the second image 221 and the optical axis-shifted image 222 obtained by optically shifting the second image 221 so that it aligns with the optical axis of the first sensor 501. For example, the image synthesis unit 13 rotates the second image 221 by 180 degrees in order to perform optical axis shifting to align the second image 221 with the optical axis of the first sensor 501. As a result, the image synthesis unit 13 converts the second image 221 into the optical axis-shifted image 222.

[0040] Figure 8 shows the third image 230 synthesized by the image synthesis unit 13. For example, the third image 230 shown in Figure 8 is an image synthesized by the image synthesis unit 13 using the projected image 216 obtained by the image conversion unit 12 shown in Figure 6 projecting the first image 211, and the optical axis converted image 222 obtained by optical axis converting the second image 221 to align with the optical axis of the first sensor 501, to create a third image that includes the user's body and wrist. In this way, the image synthesis unit 13 can synthesize images acquired from the first sensor 501 and the second sensor 502 mounted in the vehicle interior into an image that includes the user's body and wrist.

[0041] Returning to Figure 5, let's continue the explanation. The second image acquisition unit 14 identifies a plurality of first feature points corresponding to the body from the third image. The plurality of first feature points corresponding to the body include a plurality of third feature points corresponding to the wrist. Based on the plurality of third feature points corresponding to the wrist, the second image acquisition unit 14 acquires a fourth image including the hand. Specifically, the second image acquisition unit 14 identifies a plurality of first feature points corresponding to the user's body from the third image including the user's body and wrist, which has been synthesized by the image synthesis unit 13. Based on the plurality of third feature points corresponding to the wrist included in the plurality of first feature points corresponding to the body, the second image acquisition unit 14 acquires a fourth image including the user's hand.

[0042] The detection unit 15 identifies multiple second feature points corresponding to the hand from the fourth image and detects the user's hand movement based on the multiple second feature points. Specifically, the detection unit 15 identifies multiple second feature points corresponding to the user's hand from the fourth image, which includes the user's hand and was acquired by the second image acquisition unit 14, and detects the user's hand movement based on the multiple second feature points. In this embodiment, the user's hand refers to the part of the user's hand from the wrist to the fingertips.

[0043] Here, the fourth image, which includes the user's hand, will be explained using Figure 9. Figure 9 is a schematic diagram illustrating the processing of the control device according to the first embodiment. In Figure 9, the processing performed by the second image acquisition unit 14 and the detection unit 15 of the control unit 10 of the control device 1 will be explained.

[0044] Figure 9 shows the process flow in which the control device 1 identifies multiple first feature points corresponding to the user's body from a third image including the user's body and wrist, acquires a fourth image including the user's hand based on multiple third feature points corresponding to the wrist included in the multiple first feature points, identifies multiple second feature points corresponding to the user's hand from the acquired fourth image including the user's hand, and detects the movement of the user's hand based on the multiple second feature points.

[0045] For example, the second image acquisition unit 14 identifies a plurality of first feature points 231 corresponding to the user's body from the third image 230, which includes the user's body and wrist, synthesized by the image synthesis unit 13. The second image acquisition unit 14 also identifies a plurality of third feature points 233 corresponding to the user's wrist. The plurality of third feature points 233 are included in the plurality of first feature points 231. The second image acquisition unit 14 identifies where the plurality of first feature points 231 are located in the optical axis conversion image 222 and adds the plurality of first feature points 231 to the optical axis conversion image 222. Furthermore, the second image acquisition unit 14 converts the optical axis conversion image 222 to align with the optical axis of the second sensor 502. In other words, the second image acquisition unit 14 converts the optical axis conversion image 222 into a second image 234.

[0046] The second image acquisition unit 14 then acquires a fourth image 235, which includes the user's hand, from the second image 234 based on a plurality of third feature points 233 corresponding to the user's wrist. The detection unit 15 then identifies a plurality of second feature points 236 corresponding to the user's hand from the fourth image 235, which includes the user's hand, acquired by the second image acquisition unit 14, and detects the user's hand movement based on the plurality of second feature points 236. As a result, the control device 1 can detect the user's hand movement, thereby improving the accuracy of detecting the user's hand movement.

[0047] Returning to Figure 5, let's continue the explanation. The memory unit 30 stores various types of information. The memory unit 30 is implemented by hardware for storing information (in other words, data), such as memory or storage. Specifically, the memory unit 30 stores first coordinate information 31, first image information 32, second image information 33, projection transformation image information 34, third image information 35, optical axis transformation image information 36, first feature point information 37, fourth image information 38, and second feature point information 39.

[0048] The first coordinate information 31 includes the three-dimensional coordinates of the installation position of the sensor 500 having the first sensor 501 and the second sensor 502, information on the mounting angle of the sensor 500, and the three-dimensional coordinates of the position of the operating device 600. The first image information 32 includes an image of the user's body captured by the first sensor 501, and the date and time the first sensor 501 captured the image. The second image information 33 includes an image of the user's wrist captured by the second sensor 502, and the date and time the first sensor 501 captured the image.

[0049] The projection transformation image information 34 includes a projection transformation image obtained by the control device 1 in which the first image is projected to correspond to the coordinate system of the second camera. The third image information 35 includes a third image, which includes the user's body and wrist, synthesized by the control device 1 using the first and second images. The optical axis transformation image information 36 includes an optical axis transformation image obtained by the control device 1 in which the second image is optically transformed to align with the optical axis of the first sensor 501.

[0050] The first feature point information 37 is information that includes multiple first feature points corresponding to the user's body identified by the control device 1 from the third image. The first feature points include multiple third feature points corresponding to the wrist. The fourth image information 38 is information that includes the fourth image, which includes the user's hand, acquired by the control device 1. The second feature point information 39 is information that includes multiple second feature points 236 corresponding to the user's hand, detected by the control device 1 from the fourth image.

[0051] Next, the processing procedure of the control device 1 according to the first embodiment will be explained using Figure 10. Figure 10 is a flowchart of the processing procedure of the control device 1 according to the first embodiment. This process will explain the flow of the process for detecting the user's hand.

[0052] First, the first image acquisition unit 11 acquires a first image including the user's body captured by the first sensor 501 and a second image including the user's wrist captured by the second sensor 502 (step S101). Next, the image conversion unit 12 converts the first image acquired by the first image acquisition unit 11 into a projected image that corresponds to the coordinate system of the second camera (step S102).

[0053] Next, the image synthesis unit 13 uses the first image and the second image acquired by the first image acquisition unit 11 to synthesize them into a third image that includes the user's body and wrist (step S103). For example, the image synthesis unit 13 uses the projected image obtained by the image conversion unit 12 projecting the first image and the optical axis converted image obtained by optically converting the second image so that it aligns with the optical axis of the first sensor 501 to synthesize them into a third image that includes the user's body and wrist.

[0054] Next, the second image acquisition unit 14 identifies a plurality of first feature points corresponding to the user's body from the third image, which includes the user's body and wrist, synthesized by the image synthesis unit 13 (step S104). Then, the second image acquisition unit 14 acquires a fourth image, which includes the user's hand, based on a plurality of third feature points corresponding to the wrist included in the plurality of first feature points (step S105). Subsequently, the detection unit 15 identifies a plurality of second feature points corresponding to the user's hand from the fourth image, which includes the user's hand, acquired by the second image acquisition unit 14, and detects the user's hand movement based on the plurality of second feature points (step S106). When the processing in step S106 is completed, the control device 1 performs a process to control the operating device 600 based on the user's hand movement, for example.

[0055] As described above, the control device 1 according to one aspect of this disclosure acquires a first image including the user's body captured by a first camera and a second image including the user's wrist captured by a second camera. The control device 1 then combines the first and second images to form a third image including the body and wrist. Furthermore, the control device 1 identifies a plurality of first feature points corresponding to the body from the third image and acquires a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. The control device 1 then identifies a plurality of second feature points corresponding to the hand from the fourth image and detects hand movement based on the plurality of second feature points.

[0056] Furthermore, the control device 1 transforms the first image into a projected image that corresponds to the coordinate system of the second camera, and uses the projected image and an optical axis transformed image obtained by optically transforming the second image so that it aligns with the optical axis of the first camera to synthesize them into a third image.

[0057] For example, a camera mounted inside a vehicle may not be able to capture both the user's body and hands within its field of view, resulting in the acquired image not including the user's body or hands. Even in such cases, the control device 1 identifies multiple first feature points corresponding to the user's body from a composite image created by combining an image including the user's body and an image including the user's wrist. It then acquires an image including the user's hand from a third feature point corresponding to the wrist included in the identified first feature points, and identifies multiple second feature points corresponding to the user's hand from the image including the user's hand. As a result, the control device 1 can detect the movement of the user's hand from the identified second feature points. Therefore, the control device 1 can improve the accuracy of detecting the user's hand movement.

[0058] The embodiments described above can also be modified and implemented as appropriate by changing some of the configurations or functions of each of the devices described above. Therefore, several modifications of the embodiments described above will be described below as other embodiments. In the following, we will mainly describe the differences from the embodiments described above, and will omit detailed explanations of points that are common with what has already been described.

[0059] (Second Embodiment) In the embodiment described above, the control device 1 identifies a plurality of first feature points corresponding to the user's body from a third image including the user's body and wrist, and acquires a fourth image including the user's hand based on the plurality of third feature points corresponding to the wrist. For example, in the third image including the user's body and wrist synthesized by the image synthesis unit 13 of the control unit 10 of the control device 1, the first feature points of the user's body may not be identified. This phenomenon may occur, for example, when a monocular camera with a narrow field of view is used as the first sensor 501 mounted inside the vehicle.

[0060] Therefore, in the second embodiment, if the control device 1 cannot identify a plurality of first feature points corresponding to the user's body from a third image including the user's body and wrist, a configuration will be described in which the wrist is identified from a second image including the user's wrist, and a fourth image including the user's hand is obtained.

[0061] Figure 11 is a diagram showing the schematic configuration of an information system including a control device 1, which is an information processing device according to the second embodiment. The information processing device of the second embodiment further implements a first determination unit 16 of the control unit 10 of the control device 1, compared to the information processing device of the first embodiment.

[0062] The first determination unit 16 determines whether multiple first feature points corresponding to the body can be identified from the third image, which includes the user's body and wrists. Specifically, the first determination unit 16 determines whether multiple first feature points corresponding to the body can be identified from the third image, which includes the user's body and wrists, synthesized by the image synthesis unit 13.

[0063] If the first determination unit 16 determines that multiple first feature points corresponding to the body can be identified from the third image, which includes the user's body and wrist, synthesized by the image synthesis unit 13, the second image acquisition unit 14 identifies the multiple first feature points corresponding to the body from the third image and acquires a fourth image including the hand based on the multiple third feature points corresponding to the wrist. For example, the second image acquisition unit 14 identifies the third feature point corresponding to the wrist, which is included among the multiple first feature points, and acquires a fourth image including the hand. Then, the detection unit 15 identifies multiple second feature points corresponding to the user's hand from the fourth image including the user's hand acquired by the second image acquisition unit 14, and detects the movement of the user's hand based on the multiple second feature points.

[0064] On the other hand, if the first determination unit 16 determines that it cannot identify multiple first feature points corresponding to the body from the third image including the user's body and wrist, which has been synthesized by the image synthesis unit 13, the second image acquisition unit 14 fixes the target area of ​​the second image including the user's wrist acquired by the first image acquisition unit 11 and acquires a fourth image including the user's hand. Then, the first determination unit 16 determines whether it can identify multiple third feature points corresponding to the wrist from the fourth image including the hand. Specifically, the first determination unit 16 determines whether it can identify multiple third feature points corresponding to the wrist from the fourth image including the hand acquired by the second image acquisition unit 14.

[0065] If the first determination unit 16 determines that multiple third feature points corresponding to the wrist can be identified from the fourth image including the hand, the detection unit 15 identifies multiple second feature points corresponding to the user's hand from the fourth image including the user's hand acquired by the second image acquisition unit 14, and detects the user's hand movement based on the multiple second feature points.

[0066] Furthermore, if the first determination unit 16 determines that it cannot identify multiple third feature points corresponding to the wrist from the fourth image including the hand, the second image acquisition unit 14 slides the target area of ​​the second image including the user's wrist that was acquired by the first image acquisition unit 11. Then, the second image acquisition unit 14 fixes the target area of ​​the second image including the user's wrist that was acquired by the first image acquisition unit 11 and acquires the fourth image.

[0067] Thereafter, until the first determination unit 16, as described above, determines that multiple third feature points corresponding to the wrist can be identified from the fourth image including the hand, the second image acquisition unit 14 slides the target area of ​​the second image including the user's wrist acquired by the first image acquisition unit 11, fixes the slid target area of ​​the second image, and acquires the fourth image.

[0068] Here, the process by which the second image acquisition unit 14 identifies the wrist from the second image, which includes the user's wrist, will be explained using Figure 12. Figure 12 is a schematic diagram for explaining the process of the control device 1 according to the second embodiment. Figure 12 shows the second image 241. The second image acquisition unit 14 processes along the arrows shown in Figure 12 to identify the wrist from the second image. Figure 12 also shows target regions 242, 243, 244, and 245 corresponding to the wrist.

[0069] For example, as shown in Figure 12, the second image acquisition unit 13 slides the target area 243 from the target area 242 located at the upper left end of the second image 241 toward the right end. Once it has finished sliding to the upper right end, it moves to the target area 244 located at the left end and slides the target area 245 toward the right end again. The second image acquisition unit 13 then continues this process sequentially until it has finished sliding to the lower right end, and then returns to the upper left end. In this way, the second image acquisition unit 13 can identify the wrist from the second image which includes the user's wrist and acquire a fourth image which includes the user's hand.

[0070] Next, the processing procedure of the control device 1 according to the second embodiment will be explained using Figure 13. Figure 13 is a flowchart showing the processing procedure of the control device 1 according to the second embodiment. Note that the processing from steps S101 to S106 in the flowchart shown in Figure 13 is the same as the processing from steps S101 to S106 in the flowchart shown in Figure 10, so the explanation will be omitted.

[0071] In step S201, the first determination unit 16 determines whether multiple first feature points corresponding to the body can be identified from the third image, which includes the user's body and wrist, synthesized by the image synthesis unit 13 (step S201). If the first determination unit 16 determines that multiple first feature points corresponding to the body can be identified from the third image, which includes the user's body and wrist, synthesized by the image synthesis unit 13 (step S201: Yes), the process proceeds to step S105.

[0072] On the other hand, if the first determination unit 16 determines that it cannot identify multiple first feature points corresponding to the body from the third image including the user's body and wrist, which has been synthesized by the image synthesis unit 13 (step S201: No), the process proceeds to step S202. In step S202, the second image acquisition unit 14 fixes the target area of ​​the second image including the user's wrist acquired by the first image acquisition unit 11 and acquires a fourth image including the user's hand (step S202). Subsequently, the first determination unit 16 determines whether it can identify multiple second feature points corresponding to the hand from the fourth image including the hand (step S203).

[0073] If the first determination unit 16 determines that multiple second feature points corresponding to the hand can be identified from the fourth image including the hand (step S203: Yes), the process proceeds to step S106. On the other hand, if the first determination unit 16 determines that multiple second feature points corresponding to the hand cannot be identified from the fourth image including the hand (step S203: No), the process proceeds to step S204. In step S204, the second image acquisition unit 14 slides the target area of ​​the second image including the user's wrist acquired by the first image acquisition unit 11 (step S204). When step S204 is completed, the control device 1 proceeds to step S202 and continues the subsequent processing.

[0074] As described above, in one aspect of this disclosure, if a plurality of third feature points corresponding to the user's body cannot be identified from the third image, the control device 1 identifies the wrist from the second image including the user's wrist and acquires a fourth image. As a result, even if, for example, the feature points of the user's body cannot be identified in the synthesized third image including the user's body and wrist, the control device 1 can identify the wrist from the second image including the user's wrist and acquire a fourth image including the user's hand. Therefore, the control device 1 can improve the accuracy of detecting the user's hand movements.

[0075] (First variation) In the second embodiment described above, the process by which the control device 1 identifies the user's wrist from the second image including the user's wrist was described as sliding from the upper left corner to the lower right corner of the second image, but it is not limited to this. For example, the process by which the control device 1 identifies the user's wrist from the second image including the user's wrist may be a process of sliding from the lower left corner to the upper right corner of the second image.

[0076] Figure 14 is a schematic diagram illustrating the processing of the control device 1 according to the first modified example. Figure 14 shows the second image 251. The second image acquisition unit 14 processes along the arrows shown in Figure 14 to identify the wrist from the second image. Figure 14 also shows target regions 252, 253, 254, and 255 corresponding to the wrist.

[0077] For example, as shown in Figure 14, the second image acquisition unit 13 slides upward from the target area 252 located at the lower left end of the second image 251 to the target area 253. Once it has finished sliding to the upper left end, it moves to the target area 254 located at the lower end and slides upward again to the target area 255. The second image acquisition unit 13 then continues this process sequentially until it has finished sliding to the upper right end, and then returns to the lower left end. As a result, the control device 1 can identify the wrist from the second image which includes the user's wrist and acquire a fourth image which includes the user's hand. Therefore, the control device 1 can improve the accuracy of detecting the user's hand movements.

[0078] (Second variation) In the second embodiment and the first modification described above, the process of the control device 1 identifying the user's wrist from a second image including the user's wrist was described. In the second modification, the process in which the control device 1 has identified the user's wrist from a second image including the user's wrist, acquired a fourth image including the hand, and detected hand movement will be described, and the detected hand is lost.

[0079] Figure 15 is a schematic diagram illustrating the processing of the control device 1 according to the second modified example. Figure 15 shows the second image 261. The second image acquisition unit 14 processes along the arrow shown in Figure 15 to identify the wrist from the second image. Figure 15 also shows target regions 262 and 264 corresponding to the wrist. Here, the second image acquisition unit 14 has identified the wrist in target region 262 and acquired the fourth image 263.

[0080] For example, if the detection unit 15 loses track of the hand while detecting hand movement from the fourth image 263, the image acquisition unit 14 needs to acquire the fourth image including the hand again. Therefore, for example, the second image acquisition unit 13 slides from the target area 264 located at the upper left end of the second image 261 toward the upper right end, as shown in Figure 15, and continues processing until the fourth image including the user's hand is acquired. As a result, even if the control device 1 loses track of the hand movement, it can detect the hand movement again by acquiring the fourth image including the hand. Therefore, the control device 1 can improve the accuracy of detecting the user's hand movement.

[0081] (Third variation) Figure 16 is a schematic diagram illustrating the processing of the control device 1 according to the third modified example. Figure 16 shows the second image 271. The second image acquisition unit 14 processes along the arrow shown in Figure 16 to identify the wrist from the second image. Figure 16 also shows target regions 272 and 274 corresponding to the wrist. Here, the second image acquisition unit 14 has identified the wrist in target region 272 and acquired the fourth image 273.

[0082] For example, if the detection unit 15 loses track of the hand while detecting hand movement from the fourth image 273, the second image acquisition unit 13 slides to the left relative to the target area 272, as shown in Figure 16, that is, slides to the state of target area 274, which is immediately before target area 272, and continues processing until a fourth image including the user's hand can be acquired. As a result, even if the control device 1 loses track of the hand movement, it can detect the hand movement again by acquiring a fourth image including the hand. Therefore, the control device 1 can improve the accuracy of detecting the user's hand movement.

[0083] (Third embodiment) In the embodiments described above, the process by which the control device 1 identifies the user's wrist from a second image including the user's wrist was described as identifying the wrist in all areas of the second image, but the invention is not limited to this. For example, the control device 1 may identify a third feature point corresponding to the wrist from within a predetermined set area of ​​the second image and acquire a fourth image.

[0084] Figure 17 is a schematic diagram illustrating the processing of the control device 1 according to the third embodiment. Figure 17 shows the second image 281. As shown in the second image 281, the user's wrist is only present in a portion of the area in the second image 281. Therefore, the second image acquisition unit 14 identifies the third feature point corresponding to the wrist from within a predetermined set area in the second image 281 and acquires the fourth image.

[0085] For example, as shown in Figure 17, the second image acquisition unit 14 slides from the target area 282 located at the upper left end of the second image 281 to the upper right end, and then to the target area 283, in the upper half region R1 of the second image 281, and continues processing until a fourth image including the user's hand is acquired. Furthermore, the predetermined set area is not limited to the upper half of the second image 281, but may be set according to the range of motion of the user's hand. This allows the control device 1 to shorten the processing time until a fourth image including the user's hand is acquired.

[0086] (Fourth Embodiment) For example, the control device 1 may be unable to identify multiple second feature points corresponding to the hand if the fourth image, which includes the user's hand, is unclear. Therefore, the processing in the case where the fourth image, which includes the user's hand, is unclear will be described.

[0087] Figure 18 is a diagram showing the schematic configuration of an information system including a control device 1, which is an information processing device according to the fourth embodiment. The information processing device of the fourth embodiment further includes a second determination unit 17 and a correction unit 18 of the control unit 10 of the control device 1, and brightness value information 40 of the storage unit 30 of the control device 1, compared to the information processing device of the first embodiment.

[0088] The second determination unit 17 determines whether the brightness value of the fourth image satisfies predetermined conditions. Specifically, the second determination unit 17 determines whether the brightness value of the fourth image, including the user's hand, acquired by the second image acquisition unit 14 satisfies predetermined conditions. If the second determination unit 17 determines that the brightness value of the fourth image, including the user's hand, acquired by the second image acquisition unit 14 satisfies predetermined conditions, the detection unit 15 identifies a plurality of second feature points corresponding to the user's hand from the fourth image, including the user's hand, acquired by the second image acquisition unit 14, and detects the movement of the user's hand based on the plurality of second feature points.

[0089] The correction unit 18 corrects the brightness value of the fourth image if the brightness value of the fourth image does not meet predetermined conditions. Specifically, the correction unit 18 corrects the brightness value of the fourth image if the second determination unit 17 determines that the brightness value of the fourth image, including the user's hand, acquired by the second image acquisition unit 14 does not meet predetermined conditions.

[0090] Here, the correction process performed by the correction unit 18 will be described. Figure 19 is a schematic diagram illustrating the process of the control device 1 according to the fourth embodiment. Figure 19 shows the fourth image 291 before correction and the fourth image 292 after correction.

[0091] For example, if the brightness value of the fourth image 291 does not meet a predetermined condition, the correction unit 18 generates a fourth image 292 with corrected brightness values ​​of the fourth image. Here, the correction unit 18, for example, adds up the brightness of each pixel of the fourth image 291, and if the sum of the added values ​​is greater than or equal to a predetermined brightness value, it corrects the brightness value to decrease. Also, if the sum of the added values ​​is less than or equal to a predetermined brightness value, the correction unit 18 corrects the brightness value to increase. The process of correcting the brightness value is, for example, a known dynamic range expansion process. As a result, the control device 1 can make the fourth image, including the user's hand, clearer.

[0092] Returning to Figure 18, let's continue the explanation. The luminance value information 40 is information that includes the predetermined luminance value described above. The predetermined conditions for the luminance value are set according to the specifications of the sensor 500.

[0093] Next, the processing procedure of the control device 1 according to the fourth embodiment will be explained using Figure 20. Figure 20 is a flowchart showing the processing procedure of the control device 1 according to the fourth embodiment. Note that the processing from steps S101 to S106 in the flowchart shown in Figure 20 is the same as the processing from steps S101 to S106 in the flowchart shown in Figure 10, so the explanation will be omitted.

[0094] In step S301, the second determination unit 17 determines whether the brightness value of the fourth image, including the user's hand, acquired by the second image acquisition unit 14 satisfies a predetermined condition (step S301). If the second determination unit 17 determines that the brightness value of the fourth image, including the user's hand, acquired by the second image acquisition unit 14 satisfies the predetermined condition (step S301: Yes), the process proceeds to step S106.

[0095] On the other hand, if the second determination unit 17 determines that the brightness value of the fourth image, including the user's hand, acquired by the second image acquisition unit 14 does not meet a predetermined condition (step S301: No), the process proceeds to step S302. In step S302, the correction unit 18 corrects the brightness value of the fourth image (step S302). When step S302 is completed, the control device 1 proceeds to step S106.

[0096] As described above, the control device 1 according to one aspect of this disclosure corrects the brightness value of the fourth image if the brightness value of the fourth image does not meet predetermined conditions. As a result, the fourth image, including the user's hand, becomes clearer, and the control device 1 can detect the movement of the user's hand, thereby improving the accuracy of detecting the user's hand movement.

[0097] (Fourth variation) As a fourth modification, an information processing device with one sensor 500 is described. For example, in the above-described embodiment, images from two cameras were combined, but in the fourth modification, detection can be performed with just one camera by combining a still image including the user's body with a camera image showing the user's wrist and hand. Furthermore, the second and third embodiments described above, and the first, second and third modifications, describe a configuration in which images captured using the first sensor 501 and the second sensor 502 are combined into a third image including the user's body and wrist, and if multiple first feature points corresponding to the user's body cannot be identified from the combined third image, the target area is slid and determined from the second image including the user's wrist, a fourth image including the user's hand is obtained, second feature points corresponding to the hand are identified, and the movement of the user's hand is detected.

[0098] Since the sensor 500 in the fourth modified example is a single unit, depending on the position of the sensor 500 installed in the vehicle interior, for example, even if the user's wrist and hand can be imaged, the body may not be imaged. In this case, the information processing device in the fourth modified example is applied in a manner that identifies the wrist from an image including at least the user's wrist captured by one camera, acquires an image including the user's hand from the identified wrist, and detects hand movement from the image including the user's hand. Similarly, the fourth embodiment described above can also be applied to the information processing device in the fourth modified example.

[0099] For example, the information processing device according to the fourth modified example is an information processing device that operates based on the movement of a user's hand, and may include: a first image acquisition unit 11 that acquires an image of the user's wrist captured by a camera, which includes at least the user's wrist; a second image acquisition unit 14 that identifies a plurality of third feature points corresponding to the wrist from the image including the wrist and acquires an image including the user's hand based on the plurality of third feature points; and a detection unit 15 that identifies a plurality of second feature points corresponding to the user's hand from the image including the hand and detects hand movements based on the plurality of second feature points.

[0100] As a result, the control device 1 can identify the user's wrist from an image that includes the user's wrist and acquire an image that includes the user's hand. Therefore, even with only one sensor 500, the control device 1 can improve the accuracy of detecting the user's hand movements.

[0101] (Fifth embodiment) For example, the sensor 500 may be mounted inside the vehicle cabin in a configuration other than the arrangement D2 shown in Figure 3 above. Therefore, the arrangement of the sensor 500 inside the vehicle cabin will be described below.

[0102] Figure 21 is a schematic diagram illustrating the arrangement of the sensor 500 according to the fifth embodiment. The schematic diagram shown in Figure 21 shows the arrangement state D3 when the sensor 500 is mounted on the vehicle in a position where it can capture the image 201 shown in Figure 2. The sensor 500 according to the fifth embodiment is positioned in area R2 in arrangement state D3. Area R2 is an area located in front of and directly in front of the user inside the vehicle cabin.

[0103] Figure 22 is a schematic diagram illustrating the arrangement of the sensor 500 according to the fifth embodiment. The schematic diagram shown in Figure 22 shows the first sensor 501 and the second sensor 502 arranged in region R2 in the arrangement state D3 shown in Figure 21.

[0104] The first sensor 501 is positioned in the positive direction of the Z axis on the YZ plane to capture an image including the user's face and body, for example. The second sensor 502 is positioned in the negative direction of the Y axis on the YZ plane to capture an image including the user's arm, wrist, and hand, for example.

[0105] Figure 23 is a schematic diagram illustrating the processing content of the control device 1 according to the fifth embodiment. Using Figure 23, the processing performed by the image synthesis unit 13 of the control unit 10 of the control device 1 will be explained. Figure 23 shows a third image 303 synthesized by the image synthesis unit 13. For example, the third image 303 shown in Figure 23 is an image synthesized from the image 301 captured by the first sensor 501 shown in Figure 22 and the image 302 captured by the second sensor 502 shown in Figure 22.

[0106] Furthermore, images 301 and 302 are images taken so that the optical axis of the first sensor 501 and the optical axis of the second sensor 502 intersect. As a result, the image synthesis unit 13 can synthesize the images acquired from the first sensor 501 and the second sensor 502, which are mounted inside the vehicle, into an image that includes the user's body and wrist.

[0107] (Fifth variation) The fifth modification describes an arrangement of the sensor 500 in the vehicle interior that differs from that of the fifth embodiment. Figure 24 is a schematic diagram illustrating the arrangement of the sensor 500 according to the fifth modification. The schematic diagram shown in Figure 24 shows the first sensor 501 and the second sensor 502 arranged in region R2 in the arrangement state D3 shown in Figure 21. The first sensor 501 is arranged on the YZ plane in the negative direction of the Y axis and the positive direction of the Z axis, for example, to capture an image including the user's face.

[0108] The second sensor 502 is positioned on the YZ plane, facing in the negative direction of the Y axis and the negative direction of the Z axis, for example, to capture an image including the user's body, arms, wrists, and hands.

[0109] Figure 25 is a schematic diagram illustrating the processing content of the control device 1 according to the fifth modified example. Using Figure 25, the processing performed by the image synthesis unit 13 of the control unit 10 of the control device 1 will be explained.

[0110] Figure 25 shows the third image 313 synthesized by the image synthesis unit 13. For example, the third image 313 shown in Figure 25 is an image synthesized from image 311 captured by the first sensor 501 shown in Figure 24 and image 312 captured by the second sensor 502 shown in Figure 24. Furthermore, images 311 and 312 were captured so that the optical axes of the first sensor 501 and the second sensor 502 intersect. As a result, the image synthesis unit 13 can synthesize images acquired from the first sensor 501 and the second sensor 502 mounted in the vehicle interior into an image that includes the user's body and wrist.

[0111] (Sixth variation) The sixth modification describes an arrangement of the sensor 500 inside the vehicle interior that differs from the fifth embodiment and the fifth modification. Figure 26 is a schematic diagram illustrating the arrangement of the sensor 500 according to the sixth modification. The schematic diagram shown in Figure 26 shows the first sensor 501 and the second sensor 502 arranged in region R2 in the arrangement state D3 shown in Figure 21. The first sensor 501 is arranged on the YZ plane in the negative direction of the Y axis and the positive direction of the Z axis, for example, to capture an image including the user's face, body, and arms.

[0112] The second sensor 502 is positioned on the YZ plane, facing in the negative direction of the Y axis and the negative direction of the Z axis, for example, to capture an image including the user's wrist and hand. Figure 27 is a schematic diagram illustrating the processing content of the control device 1 according to the sixth modified example. Using Figure 27, the processing performed by the image synthesis unit 13 of the control unit 10 of the control device 1 will be explained.

[0113] Figure 27 shows the third image 323 synthesized by the image synthesis unit 13. For example, the third image 323 shown in Figure 27 is an image synthesized from image 321 captured by the first sensor 501 shown in Figure 26 and image 322 captured by the second sensor 502 shown in Figure 26. Furthermore, images 321 and 322 were captured so that the optical axes of the first sensor 501 and the second sensor 502 do not intersect. As a result, the image synthesis unit 13 can synthesize images acquired from the first sensor 501 and the second sensor 502 mounted in the vehicle interior into an image that includes the user's body and wrist.

[0114] (Sixth Embodiment) In the first embodiment described above, the control device 1 synthesizes a third image including the user's body and wrist, identifies a plurality of first feature points corresponding to the body from the third image, and uses a second image corresponding to a plurality of third feature points corresponding to the wrist to obtain a fourth image including the user's hand. As a sixth embodiment, a configuration will be described in which a fourth image including the user's hand is obtained using a second image including the user's wrist, without synthesizing a third image including the user's body and wrist.

[0115] Figure 28 is a diagram showing the schematic configuration of an information system including a control device 1, which is an information processing device according to the sixth embodiment. The information processing device according to the sixth embodiment comprises a sensor 500, an operating device 600, and a control device 1. The control device 1 comprises a control unit 10 and a storage unit 30. The control unit 10 comprises a first image acquisition unit 11, a second image acquisition unit 14, and a detection unit 15. The same details as in the first embodiment will be omitted from the explanation.

[0116] For example, the image captured by the first sensor 501 may include the user's body and wrist. In this case, the control device 1 can identify the user's wrist from the image containing the user's body and wrist. The specific processing of the functional unit will be described below.

[0117] The first image acquisition unit 11 acquires a fifth image, which includes the user's body and wrist, captured by the first sensor 501, and a second image, which includes the user's wrist, captured by the second camera.

[0118] The second image acquisition unit 14 identifies multiple third feature points corresponding to the wrist from the fifth image, converts these multiple third feature points into coordinates in the second image, and acquires a fourth image including the user's hand based on the multiple third feature points in the second image. The coordinate-transformed image will be explained using Figure 29.

[0119] Figure 29 is a schematic diagram illustrating the processing content of the control device 1 according to the sixth embodiment. Figure 29 shows a fifth image 331 captured by the first sensor 501, including the user's body and wrist, and the coordinate points 332 of the four corners when the hand is moved forward left and right and backward left and right, and a second image 333 captured by the second camera, including the user's wrist, and the coordinate points 334 of the four corners when the hand is moved forward left and right and backward left and right.

[0120] Here, the fifth image 331 includes the user's body and wrist, but only a portion of the user's hand. In other words, the fifth image 331 does not include the entire user's hand. Therefore, it is not possible to identify multiple second feature points corresponding to the user's hand from the fifth image 331 alone. On the other hand, the second image 333 includes the user's wrist and the user's hand.

[0121] First, the second image acquisition unit 14 identifies multiple third feature points corresponding to the wrist from the fifth image 331 acquired by the first image acquisition unit 11. The second image acquisition unit 14 also transforms the coordinates of the multiple third feature points in the fifth image 331 to the coordinates of the second image 333 by projective transformation using the coordinate point 332 of the fifth image 331 and the coordinate point 334 of the second image 333. Then, the second image acquisition unit 14 identifies multiple third feature points in the second image 333 and acquires a fourth image including the user's hand based on the multiple third feature points in the second image 333.

[0122] As a result, if the image acquired by the first image acquisition unit 11 includes the user's body and wrist, the second image acquisition unit 14 can acquire an image including the user's hand from the image captured by the second sensor 502 without combining the images captured by the first sensor 501 and the second sensor 502.

[0123] Returning to Figure 28, let's continue the explanation. The memory unit 30 stores the first coordinate information 31, the second image information 33, the first feature point information 37, the fourth image information 38, the second feature point information 39, the fifth image information 41, and the coordinate transformation image information 42.

[0124] The fifth image information 41 includes an image of the user's body and wrist captured by the first sensor 501, and the date and time the first sensor 501 captured the image. The coordinate transformation image information 42 includes a coordinate transformation image obtained by the second image acquisition unit 14 by projecting the fifth image onto the coordinate system of the second camera.

[0125] Next, the processing procedure of the control device 1 according to the sixth embodiment will be explained using Figure 30. Figure 30 is a flowchart showing the processing procedure of the control device 1 according to the sixth embodiment. Note that the processing of step S105 in the flowchart shown in Figure 30 is the same as the processing of step S105 in the flowchart shown in Figure 10, so the explanation will be omitted.

[0126] In step S401, the first image acquisition unit 11 acquires a fifth image including the user's body and wrist captured by the first sensor 501, and a second image including the user's wrist captured by the second sensor 502 (step S401). In step S402, the second image acquisition unit 14 identifies a plurality of third feature points corresponding to the wrist from the fifth image 331 acquired by the first image acquisition unit 11 (step S402).

[0127] In step S403, the second image acquisition unit 14 transforms the coordinates of multiple third feature points in the fifth image 331 to the coordinates of the second image 333 by projective transformation using the coordinate point 332 of the fifth image 331 and the coordinate point 334 of the second image 333 (step S403). In step S404, the second image acquisition unit 14 identifies multiple third feature points in the second image 333 and acquires a fourth image including the user's hand based on the multiple third feature points in the second image 333 (step S404). When step S404 is completed, the control device 1 proceeds to step S106.

[0128] As described above, the control device 1 according to one aspect of this disclosure acquires a fifth image including the user's body and wrist captured by the first camera, and a second image including the user's wrist captured by the second camera. The control device 1 also identifies a plurality of third feature points corresponding to the wrist from the coordinate transformation image obtained by projecting the fifth image onto the coordinate system of the second camera, and acquires a fourth image including the user's hand using the second image corresponding to the plurality of third feature points. Furthermore, the control device 1 identifies a plurality of second feature points corresponding to the hand from the fourth image and detects hand movement based on the plurality of second feature points.

[0129] Even if the acquired image only contains the user's body and not the hand, the control device 1 identifies multiple third feature points corresponding to the wrist from the image including the user's body and wrist, acquires an image including the user's hand from the identified multiple third feature points, and identifies multiple second feature points corresponding to the user's hand from the image including the user's hand. As a result, the control device 1 can detect the movement of the user's hand from the identified multiple second feature points. Therefore, the control device 1 can improve the accuracy of detecting the movement of the user's hand.

[0130] While embodiments of this disclosure have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0131] Furthermore, the notation "...part" in the above-described embodiment may be replaced with other notations such as "...circuitry," "...assembly," "...device," "...unit," or "...module."

[0132] In the embodiments described above, the disclosure has been explained using examples configured with hardware, but the disclosure can also be implemented with software in conjunction with hardware.

[0133] Furthermore, each functional block used in the description of the above embodiments is typically implemented as an integrated circuit (LSI). The integrated circuit controls each functional block used in the description of the above embodiments and may have input and output terminals. These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Here, we refer to it as an LSI, but depending on the degree of integration, it may also be called an IC, system LSI, super LSI, or ultra LSI.

[0134] Furthermore, the method of integrated circuit implementation is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors and memory. After LSI manufacturing, FPGAs (Field Programmable Gate Arrays) that can be programmed, or reconfigurable processors that allow for the reconfiguration of the connections or settings of circuit cells inside the LSI, may also be used.

[0135] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that can replace LSIs, then naturally, functional blocks could be integrated using those technologies. The application of biotechnology, for example, is a possibility.

[0136] Furthermore, the effects described in the embodiments herein are merely illustrative and not limiting, and other effects may also occur.

[0137] (Note) Based on the above description of embodiments, the following technologies are disclosed. (Technology 1) An information processing device that operates based on the user's hand movements, A first image acquisition unit acquires a first image including the user's body captured by the first camera and a second image including the user's wrist captured by the second camera. An image synthesis unit that synthesizes the first and second images into a third image including the body and the wrist, A second image acquisition unit identifies a plurality of first feature points corresponding to the body from the third image and acquires a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. A detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image and detects the movement of the hand based on the plurality of second feature points, An information processing device equipped with the following features. (Technology 2) The system includes an image transformation unit that transforms the first image into a projected image that corresponds to the coordinate system of the second camera, The image synthesis unit synthesizes the projection transformation image and the optical axis transformation image obtained by optically transforming the second image so as to align it with the optical axis of the first camera into the third image. The information processing device described in Technology 1 above. (Technology 3) If the plurality of first feature points cannot be identified from the third image, the second image acquisition unit identifies the third feature point corresponding to the wrist from the second image and acquires the fourth image. The information processing device described in Technology 2 above. (Technology 4) The second image acquisition unit identifies a third feature point corresponding to the wrist from within a preset area of ​​the second image and acquires the fourth image. The information processing device described in Technology 3 above. (Technology 5) An information processing device that operates based on the user's hand movements, A first image acquisition unit acquires a fifth image including the user's body and wrist captured by the first camera, and a second image including the user's wrist captured by the second camera. A second image acquisition unit identifies a plurality of third feature points corresponding to the wrist from a coordinate transformation image obtained by projecting the fifth image onto the coordinate system of the second camera, and uses the second image corresponding to the plurality of third feature points to acquire a fourth image including the user's hand. A detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image and detects the movement of the hand based on the plurality of second feature points, An information processing device equipped with the following features. (Technology 6) If the brightness value of the fourth image does not meet predetermined conditions, the system includes a correction unit that corrects the brightness value of the fourth image. An information processing device as described in any one of the above technologies 1 to 5. (Technology 7) An information processing device that operates based on the user's hand movements, An image acquisition unit that acquires an image including the user's wrist captured by the camera, A second image acquisition unit identifies a plurality of third feature points corresponding to the wrist from the image including the wrist, and acquires an image including the user's hand based on the plurality of third feature points. A detection unit identifies a plurality of second feature points corresponding to the user's hand from the image including the hand, and detects the movement of the hand based on the plurality of second feature points. An information processing device equipped with the following features. (Technology 8) The second image acquisition unit identifies the wrist from within a preset area of ​​the image including the wrist and acquires an image including the hand. The information processing device described in Technology 7 above. (Technology 9) The system includes a correction unit that corrects the brightness value of the image including the hand if the brightness value of the image including the hand does not meet predetermined conditions. The information processing device described in technology 7 or 8 above. (Technology 10) An information processing system comprising a camera and an information processing device that operates based on the user's hand movements, The camera comprises a first camera and a second camera. The aforementioned information processing device is A first image acquisition unit acquires a first image including the user's body captured by the first camera and a second image including the user's wrist captured by the second camera. An image synthesis unit that synthesizes the first and second images into a third image including the body and the wrist, A second image acquisition unit identifies a plurality of first feature points corresponding to the body from the third image and acquires a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. A detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image and detects the movement of the hand based on the plurality of second feature points, An information processing system equipped with the following features. (Technology 11) An information processing method performed by an information processing device that operates based on the movements of a user's hand, A first image acquisition step involves acquiring a first image including the user's body captured by a first camera, and a second image including the user's wrist captured by a second camera. Image synthesis step of combining the first image and the second image to form a third image including the body and the wrist, A second image acquisition step involves identifying a plurality of first feature points corresponding to the body from the third image and acquiring a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. A detection step involves identifying a plurality of second feature points corresponding to the hand from the fourth image and detecting the movement of the hand based on the plurality of second feature points. Information processing methods including (Technology 12) On the computer, A first image acquisition step involves acquiring a first image including the user's body captured by a first camera, and a second image including the user's wrist captured by a second camera. Image synthesis step of combining the first image and the second image to form a third image including the body and the wrist, A second image acquisition step involves identifying a plurality of first feature points corresponding to the body from the third image and acquiring a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. A detection step involves identifying a plurality of second feature points corresponding to the hand from the fourth image and detecting the movement of the hand based on the plurality of second feature points. An information processing program that executes [something]. [Explanation of symbols]

[0138] 1 Control device 10 Control Unit 11. First Image Acquisition Unit 12 Image conversion unit 13 Image Synthesis Unit 14. Second Image Acquisition Unit 15 Detection unit 16 1st Judgment Section 17 Second Judgment Section 18 Correction section 30 Storage section 500 sensors 501 First Sensor 502 Second Sensor 600 Operating equipment

Claims

1. An information processing device that operates based on the user's hand movements, A first image acquisition unit acquires a first image including the user's body captured by a first camera and a second image including the user's wrist captured by a second camera. An image synthesis unit that synthesizes the first and second images into a third image including the body and the wrist, A second image acquisition unit identifies a plurality of first feature points corresponding to the body from the third image and acquires a fourth image including the user's hand based on the third feature point corresponding to the wrist. A detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image and detects the movement of the hand based on the plurality of second feature points, An information processing device equipped with the following features.

2. The system includes an image transformation unit that transforms the first image into a projected image that corresponds to the coordinate system of the second camera, The image synthesis unit synthesizes the projection transformation image and the optical axis transformation image obtained by optically transforming the second image so as to align it with the optical axis of the first camera into the third image. The information processing apparatus according to claim 1.

3. If the plurality of first feature points cannot be identified from the third image, the second image acquisition unit identifies the third feature points from the second image and acquires the fourth image. The information processing apparatus according to claim 2.

4. The second image acquisition unit identifies a third feature point corresponding to the wrist from within a preset area of ​​the second image and acquires the fourth image. The information processing apparatus according to claim 3.

5. An information processing device that operates based on the user's hand movements, A first image acquisition unit acquires a fifth image including the user's body and wrist captured by the first camera, and a second image including the user's wrist captured by the second camera. A second image acquisition unit obtains a fourth image including the user's hand by using the second image corresponding to the plurality of third feature points, which are obtained by projecting the fifth image onto the coordinate system of the second camera and identifying a plurality of third feature points corresponding to the wrist from the coordinate-transformed image, A detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image and detects the movement of the hand based on the plurality of second feature points, An information processing device equipped with the following features.

6. If the brightness value of the fourth image does not meet predetermined conditions, a correction unit is provided to correct the brightness value of the fourth image. The information processing apparatus according to any one of claims 1 to 5.

7. An information processing device that operates based on the user's hand movements, An image acquisition unit that acquires an image including the user's wrist captured by the camera, A second image acquisition unit identifies a plurality of third feature points corresponding to the wrist from the image including the wrist, and acquires an image including the user's hand based on the plurality of third feature points. A detection unit identifies a plurality of second feature points corresponding to the user's hand from the image including the hand, and detects the movement of the hand based on the plurality of second feature points. An information processing device equipped with the following features.

8. The second image acquisition unit identifies the wrist from within a preset area of ​​the image including the wrist and acquires an image including the hand. The information processing apparatus according to claim 7.

9. The system includes a correction unit that corrects the brightness value of the image including the hand if the brightness value of the image including the hand does not meet predetermined conditions. The information processing apparatus according to claim 7 or 8.

10. An information processing system comprising a camera and an information processing device that operates based on the user's hand movements, The camera comprises a first camera and a second camera. The aforementioned information processing device is A first image acquisition unit acquires a first image including the user's body captured by the first camera and a second image including the user's wrist captured by the second camera. An image synthesis unit that synthesizes the first and second images into a third image including the body and the wrist, A second image acquisition unit identifies a plurality of first feature points corresponding to the body from the third image and acquires a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. A detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image and detects the movement of the hand based on the plurality of second feature points, An information processing system equipped with the following features.

11. An information processing method performed by an information processing device that operates based on the movements of a user's hand, A first image acquisition step involves acquiring a first image including the user's body captured by a first camera, and a second image including the user's wrist captured by a second camera. Image synthesis step of combining the first image and the second image to form a third image including the body and the wrist, A second image acquisition step involves identifying a plurality of first feature points corresponding to the body from the third image, and acquiring a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. A detection step involves identifying a plurality of second feature points corresponding to the hand from the fourth image and detecting the movement of the hand based on the plurality of second feature points. Information processing methods including

12. On the computer, A first image acquisition step involves acquiring a first image including the user's body captured by a first camera, and a second image including the user's wrist captured by a second camera. Image synthesis step of combining the first image and the second image to form a third image including the body and the wrist, A second image acquisition step involves identifying a plurality of first feature points corresponding to the body from the third image, and acquiring a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. A detection step involves identifying a plurality of second feature points corresponding to the hand from the fourth image and detecting the movement of the hand based on the plurality of second feature points. An information processing program that executes [something].

Images