Information processing device, information processing system, information processing method, recording medium, and computer program product

Abstract

The invention provides an information processing apparatus, an information processing system, an information processing method, a recording medium, and a computer program product. An information processing device according to one embodiment of the present disclosure is provided with a first image acquisition unit, an image synthesis unit, a second image acquisition unit, and a detection unit. A first image acquisition unit acquires a first image including a body of a user captured by a first camera, and a second image including a wrist of the user captured by a second camera. The image compositing section generates a third image including the body and the wrist by compositing the first image and the second image. The second image acquisition section determines a plurality of first feature points corresponding to the body from the third image, and acquires a fourth image including the hand of the user based on a plurality of third feature points corresponding to the wrist. The detection unit identifies a plurality of second feature points corresponding to the hand from the fourth image, and detects movement of the hand on the basis of the plurality of second feature points.

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus, an information processing system, an information processing method, and a recording medium. Background Technology

[0002] In the past, there have been various devices that detect the movement of a user's hand and perform motion control based on the detected hand movement (for example, see Patent Document 1). For example, there are known technologies where, for displays that are out of reach of the user's hand, or operating devices for users inside a vehicle to point out objects outside the vehicle, the user does not directly touch them, but changes the content displayed by the operating devices based on the movement of the user's hand.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2024-34419 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] In the aforementioned techniques, for example, as a process for detecting a user's hand movement, there is a situation where: an image captured by a camera is acquired, the user's body and hand are determined from the acquired image, and the user's hand movement is detected based on the direction the hand is pointing. However, cameras mounted inside vehicles sometimes cannot capture both the user's body and hand within the camera's field of view, thus not including the user's body and hand in the acquired image. Therefore, there is a situation where, in the process of detecting the user's hand movement, the user's body and hand cannot be determined from the acquired image, and thus the user's hand movement cannot be detected.

[0008] This disclosure provides an information processing device, information processing system, information processing method, and recording medium that can improve the accuracy of detecting a user's hand movements.

[0009] Solution for solving the problem

[0010] One aspect of this disclosure relates to an information processing apparatus comprising a first image acquisition unit, an image synthesis unit, a second image acquisition unit, and a detection unit. The first image acquisition unit acquires a first image including a user's body captured by a first camera, and a second image including the user's wrist captured by a second camera. The image synthesis unit generates a third image including the body and wrist by combining the first and second images. The second image acquisition unit determines a plurality of first feature points corresponding to the body from the third image, and acquires a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist. The detection unit determines a plurality of second feature points corresponding to the hand from the fourth image, and detects hand movement based on the plurality of second feature points.

[0011] The effects of the invention

[0012] The information processing apparatus disclosed herein can improve the accuracy of detecting a user's hand movements. Furthermore, the effects described herein are not necessarily limited to those described herein; any of the effects described in this specification may also apply. Attached Figure Description

[0013] Figure 1 This is a schematic diagram illustrating an example of an information processing device.

[0014] Figure 2 This is a schematic diagram used to illustrate the processing of an information processing device.

[0015] Figure 3 This is a schematic diagram illustrating the configuration of the sensor involved in the first comparative example.

[0016] Figure 4 This is a schematic diagram illustrating the configuration of the sensor involved in the first embodiment.

[0017] Figure 5 This is a diagram showing the outline structure of an information system including a control device of an information processing apparatus according to the first embodiment.

[0018] Figure 6 This is a schematic diagram illustrating the processing of the control device involved in the first embodiment.

[0019] Figure 7 This is a schematic diagram illustrating the processing of the control device involved in the first embodiment.

[0020] Figure 8 This is a schematic diagram illustrating the processing of the control device involved in the first embodiment.

[0021] Figure 9 This is a schematic diagram illustrating the processing of the control device involved in the first embodiment.

[0022] Figure 10 This is a flowchart illustrating the processing procedure of the control device according to the first embodiment.

[0023] Figure 11 This is a diagram showing the outline structure of an information system including a control device for an information processing apparatus as described in the second embodiment.

[0024] Figure 12 This is a schematic diagram illustrating the processing of the control device involved in the second embodiment.

[0025] Figure 13 This is a flowchart illustrating the processing procedure of the control device according to the second embodiment.

[0026] Figure 14 This is a schematic diagram illustrating the processing of the control device involved in the first modified example.

[0027] Figure 15 This is a schematic diagram illustrating the processing of the control device involved in the second variation.

[0028] Figure 16 This is a schematic diagram illustrating the processing of the control device involved in the third variation.

[0029] Figure 17 This is a schematic diagram illustrating the processing of the control device involved in the third embodiment.

[0030] Figure 18 This is a diagram showing the outline structure of an information system including a control device for an information processing apparatus as described in the fourth embodiment.

[0031] Figure 19 This is a schematic diagram illustrating the processing of the control device involved in the fourth embodiment.

[0032] Figure 20 This is a flowchart illustrating the processing procedure of the control device according to the fourth embodiment.

[0033] Figure 21 This is a schematic diagram illustrating the configuration of the sensor according to the fifth embodiment.

[0034] Figure 22 This is a schematic diagram illustrating the configuration of the sensor according to the fifth embodiment.

[0035] Figure 23 This is a schematic diagram illustrating the processing of the control device according to the fifth embodiment.

[0036] Figure 24 This is a schematic diagram illustrating the configuration of the sensor involved in the fifth variation.

[0037] Figure 25 This is a schematic diagram illustrating the processing of the control device involved in the fifth variation.

[0038] Figure 26 This is a schematic diagram illustrating the configuration of the sensor involved in the sixth variation.

[0039] Figure 27 This is a schematic diagram used to illustrate the processing of the control device involved in the sixth variation.

[0040] Figure 28 This is a diagram showing the outline structure of an information system including a control device for an information processing apparatus according to the sixth embodiment.

[0041] Figure 29 This is a schematic diagram illustrating the processing of the control device according to the sixth embodiment.

[0042] Figure 30 This is a flowchart illustrating the processing procedure of the control device according to the sixth embodiment. Detailed Implementation

[0043] Hereinafter, embodiments of the information processing apparatus involved in this disclosure will be described with reference to the accompanying drawings.

[0044] (First Implementation)

[0045] Before describing the information processing apparatus according to the first embodiment, an apparatus that operates based on the movement of a user's hand will be described.

[0046] Figure 1 This is a schematic diagram illustrating an example of an information processing device. Figure 1 An example of an information processing device that operates based on the movement of a user's hand is shown. The information processing device includes a sensor 500 and an operating device 600. The sensor 500 is, for example, a camera. The sensor 500 captures an image of the user acting as the operator. The user has an arm. In this disclosure, the arm is envisioned to include not only the upper arm and forearm, but also the hand. The hand also includes fingertips 72. The operating device 600 is, for example, a display device such as a vehicle-mounted monitor. The information processing device controls the data displayed by the operating device 600 based on the position of the user's fingertips 72.

[0047] Additionally, for example, when a user points from inside the vehicle to a building outside, the information processing device controls the operation device 600 based on the position of the user's fingertip 72, causing the operation device 600 to display the direction of the pointed object. The object pointed to by the user's fingertip 72 includes the operation device 600, buildings outside the vehicle, and other objects.

[0048] In addition, the information processing device calculates the angle for accepting the operation of the device. Specifically, the information processing device calculates the angle 900, which represents the angle formed by the vehicle's direction of travel M1 and the angle line 700 in the XZ plane where the user's fingertip 72 points to the operating device 600.

[0049] In addition, Figure 1 In the accompanying drawings relating to the operating device 600 described below, the mutually orthogonal X-axis, Y-axis, and Z-axis refer to the left-right, up-down, and front-back directions of the operating device 600, respectively. Furthermore, in the following description, when only the X-direction, Y-direction, or Z-direction is mentioned, it refers to the axial direction of each direction, and is assumed to include two directions with opposite orientations.

[0050] Additionally, when the X-axis is defined as the positive direction, it is a direction from left to right; when the Y-axis is defined as the positive direction, it is a direction from bottom to top; and when the Z-axis is defined as the positive direction, it is a direction from front to back. When the X-axis is defined as the negative direction, it is a direction from right to left; when the Y-axis is defined as the negative direction, it is a direction from top to bottom; and when the Z-axis is defined as the negative direction, it is a direction from back to front.

[0051] Figure 2 This is a schematic diagram used to illustrate the processing of an information processing device. Figure 2 The processing of the information processing device detecting the user's fingertip 72 will be explained. First, the information processing device acquires an image 201 captured by the sensor 500. Then, the information processing device determines multiple feature points 203 corresponding to the user's body and wrist from the acquired image 202.

[0052] Furthermore, the information processing device acquires an image 204 including multiple feature points 203 corresponding to the user's wrist. Then, the information processing device determines multiple feature points 205 corresponding to the user's hand from the acquired image 204 and detects the user's fingertips 72. When the information processing device detects the user's fingertips 72, it controls the data displayed on the operating device 600 based on the position of the user's fingertips 72.

[0053] Figure 3 This is a schematic diagram illustrating the configuration of the sensor 500 involved in the first comparative example. Figure 3 The schematic diagram shows the sensor 500 mounted on a device capable of capturing images. Figure 2 The configuration state D1 is shown in the virtual position of image 201. Furthermore, in this disclosure, as an example of a vehicle, [the following is used]. Figure 3 The right-hand drive vehicle shown can be replaced by a left-hand drive vehicle. Here, it is assumed that the user's hands are located at the steering wheel 721 in the vehicle.

[0054] For example, in order to shoot Figure 2 As shown in image 201, sometimes sensor 500 is located outside the vehicle, as in configuration state D1. Assuming that sensor 500 is positioned inside the vehicle, the user's body and wrist are not included in the image captured by sensor 500 due to the mismatch in the sensor 500's viewing angle.

[0055] Therefore, the information processing device cannot determine the user's wrist, and thus sometimes cannot detect the user's fingertips 72. Therefore, this embodiment provides, for example, an information processing device equipped with a sensor 500 positioned to capture images of the user's body and wrist, including inside the vehicle, thereby improving the accuracy of detecting the user's hand movements.

[0056] Figure 4 This is a schematic diagram illustrating the configuration of the sensor 500 according to the first embodiment. Figure 4 The schematic diagram shows the sensor 500 mounted on a device capable of capturing images. Figure 2 The image 201 shows the configuration state D2 of the vehicle interior location.

[0057] For example, as shown in configuration state D2, sensor 500 has a first sensor 501 and a second sensor 502. The first sensor 501 and the second sensor 502 are positioned in front of and in front of the user inside the vehicle interior. For example, the first sensor 501 captures an image including the user's body and is positioned near a rearview mirror mounted inside the vehicle interior.

[0058] Additionally, for example, the second sensor 502 captures an image, including the user's wrist, and is positioned directly above the user's seat inside the vehicle interior. By configuring the first sensor 501 and the second sensor 502 as described above, they are able to capture images of the user's body and hands inside the vehicle interior.

[0059] Figure 5 This is a diagram showing a general structure of an information system including a control device as part of the information processing apparatus according to the first embodiment. The information processing apparatus according to the first embodiment includes a sensor 500, an operating device 600, and a control device 1.

[0060] Sensor 500 is, for example, a camera device. As an example, the camera device is an infrared camera. However, the camera device is not limited to infrared cameras; it includes visible light cameras. Sensor 500 is an example of an imaging unit. Sensor 500 captures images of the user acting as an operator and outputs the captured images to control device 1.

[0061] Sensor 500 has a first sensor 501 and a second sensor 502. The first sensor 501 is also referred to as a first camera. The second sensor 502 is also referred to as a second camera. For example, the first sensor 501 captures an image including the user's body. Additionally, for example, the second sensor 502 captures an image including the user's wrist. Sensor 500 continuously performs image capture processing and outputs the image to the control device 1.

[0062] Sensor 500 is mounted inside the vehicle. Specifically, the first sensor 501 and the second sensor 502 are as follows: Figure 4 As shown, the optical axes of the first sensor 501 and the second sensor 502 are separated, with the optical axis of the first sensor 501 intersecting. For example, the first sensor 501 is located in front of the user, in the positive direction of the Y-axis and the negative direction of the Z-axis on the YZ plane. That is, the first sensor 501 is located diagonally above the user.

[0063] Additionally, for example, the second sensor 502 is positioned in the positive Y-axis and positive Z-axis directions on the YZ plane, with the user as the reference. That is, the second sensor 502 is located diagonally behind the user.

[0064] The operating device 600 is a display unit that displays various data. The operating device 600 is, for example, a display device such as a vehicle-mounted monitor. The control device 1 processes the data displayed on the operating device 600 based on the movement of the user's fingertip 72 detected by the sensor 500.

[0065] The control device 1 includes a control unit 10 and a storage unit 30. The control unit 10 is configured, for example, as a CPU (Central Processing Unit), which uniformly controls the operation of each part of the control device 1. The control device 1 according to this embodiment has ROM and RAM (not shown). ROM stores various programs. RAM is the working area for the CPU when executing programs.

[0066] The control device 1 includes, for example, a processor and a memory. The processor executes a program stored in the memory to implement the functions of the control unit 10 and its various functional blocks. A CPU is an example of a processor. The storage unit 30 is an example of a memory.

[0067] The CPU uses RAM as a working area to execute programs stored in ROM, thereby achieving... Figure 5The first image acquisition unit 11, image transformation unit 12, image synthesis unit 13, second image acquisition unit 14, and detection unit 15 are shown. It can also be referred to as a control device 1, which includes the first image acquisition unit 11, image transformation unit 12, image synthesis unit 13, second image acquisition unit 14, and detection unit 15. The first image acquisition unit 11, image transformation unit 12, image synthesis unit 13, second image acquisition unit 14, and detection unit 15 can also be implemented using different hardware.

[0068] The first image acquisition unit 11 acquires a first image including the user's body captured by the first sensor 501, and a second image including the user's wrist captured by the second sensor 502. The first image and the second image are captured respectively with the optical axis of the first sensor 501 intersecting the optical axis of the second sensor 502.

[0069] The image transformation unit 12 transforms the first image acquired by the first image acquisition unit 11 into a projective transformation image obtained by projective transformation in a manner corresponding to the coordinate system of the second camera.

[0070] Here, use Figure 6 Let's now explain the first image and the projective transformation image. Figure 6 This is a schematic diagram illustrating the processing of the control device 1 according to the first embodiment. Figure 6 The processing performed by the image transformation unit 12 of the control unit 10 of the control device 1 will be explained. Figure 6 The image shows a first image 211 of the user's body acquired by the first image acquisition unit 11, and a projective transformed image 216 obtained by the image transformation unit 12 performing a projective transformation on the first image 211.

[0071] The first image 211 shows coordinates 212, 213, 214, and 215 in the first sensor 501. The image transformation unit 12 performs a projective transformation on the first image 211 corresponding to the coordinate system in the second sensor 502, for example, projecting from coordinate 212 to coordinate 217. Similarly, the image transformation unit 12 projectes from coordinate 213 to coordinate 218, from coordinate 214 to coordinate 219, and from coordinate 215 to coordinate 220.

[0072] return Figure 5Continuing the explanation, the image compositing unit 13 generates a third image including the user's body and wrist by compositing the first image acquired by the first image acquisition unit 11 with the second image. For example, the image compositing unit 13 generates a third image including the user's body and wrist by compositing the projective transformed image 216 obtained by the image transformation unit 12 performing a projective transformation on the first image 211 with an optical axis transformed image obtained by performing an optical axis transformation on the second image in a manner consistent with the optical axis of the first sensor 501.

[0073] Here, use Figure 7 and Figure 8 To illustrate the third image, which includes the user's body and wrist. Figure 7 and Figure 8 This is a schematic diagram illustrating the processing of the control device 1 according to the first embodiment. Figure 7 and Figure 8 The processing performed by the image synthesis unit 13 of the control unit 10 of the control device 1 will be described.

[0074] Figure 7 The diagram shows a second image 221 and an optically transformed image 222 obtained by optically transforming the second image 221 in a manner aligned with the optical axis of the first sensor 501. For example, the image compositing unit 13 rotates the second image 221 by 180 degrees to perform an optical axis transformation on the second image 221 aligned with the optical axis of the first sensor 501. Thus, the image compositing unit 13 transforms the second image 221 into the optically transformed image 222.

[0075] Figure 8 The image 230 generated by the image compositing unit 13 is shown. For example, Figure 8 The third image 230 shown is created by the image compositing unit 13 through... Figure 6 The image shown is generated by combining the projective transformed image 216 obtained by the image transformation unit 12 performing a projective transformation on the first image 211 with the optical axis transformed image 222 obtained by performing an optical axis transformation on the second image 221 in a manner aligned with the optical axis of the first sensor 501. The third image 230 includes the user's body and wrist. In this way, the image synthesis unit 13 generates an image 230 that includes the user's body and wrist by combining the images acquired from the first sensor 501 and the second sensor 502 mounted in the vehicle interior.

[0076] return Figure 5Continuing the explanation, the second image acquisition unit 14 determines a plurality of first feature points corresponding to the body from the third image. These first feature points corresponding to the body include a plurality of third feature points corresponding to the wrist. Based on the plurality of third feature points corresponding to the wrist, the second image acquisition unit 14 acquires a fourth image including the hand. Specifically, the second image acquisition unit 14 determines a plurality of first feature points corresponding to the user's body from the third image including the user's body and wrist generated by the image synthesis unit 13. Based on the plurality of third feature points corresponding to the wrist included in the plurality of first feature points corresponding to the body, the second image acquisition unit 14 acquires a fourth image including the user's hand.

[0077] The detection unit 15 determines multiple second feature points corresponding to the user's hand from the fourth image including the user's hand acquired by the second image acquisition unit 14, and detects the movement of the user's hand based on the multiple second feature points. In this embodiment, the user's hand refers to the part from the user's wrist to the fingertips.

[0078] Here, use Figure 9 Let's now explain the fourth image, which includes the user's hand. Figure 9 This is a schematic diagram illustrating the processing of the control device according to the first embodiment. Figure 9 The processing performed by the second image acquisition unit 14 and the detection unit 15 of the control unit 10 of the control device 1 will be described.

[0079] Control device 1 determines multiple first feature points corresponding to the user's body from a third image including the user's body and wrist. Based on multiple third feature points corresponding to the wrist included among the multiple first feature points, control device 1 acquires a fourth image including the user's hand. Control device 1 determines multiple second feature points corresponding to the user's hand from the acquired fourth image including the user's hand. Then, control device 1 detects the movement of the user's hand based on the multiple second feature points.

[0080] The second image acquisition unit 14 determines a plurality of first feature points 231 corresponding to the user's body from the third image 230, which includes the user's body and wrist, generated by the image synthesis unit 13. Additionally, the second image acquisition unit 14 determines a plurality of third feature points 233 corresponding to the user's wrist. The plurality of third feature points 233 are included among the plurality of first feature points 231. The second image acquisition unit 14 determines the positions corresponding to the plurality of first feature points 231 in the optical axis transformation image 222, and appends the plurality of first feature points 231 to the optical axis transformation image 222 based on the determined positions. Furthermore, the second image acquisition unit 14 performs optical axis transformation on the optical axis transformation image 222 in a manner consistent with the optical axis of the second sensor 502. That is, the second image acquisition unit 14 transforms the optical axis transformation image 222 into the second image 234.

[0081] Then, the second image acquisition unit 14 acquires a fourth image 235 including the user's hand from the second image 234 based on a plurality of third feature points 233 corresponding to the user's wrist. Additionally, the detection unit 15 determines a plurality of second feature points 236 corresponding to the user's hand from the fourth image 235 including the user's hand. Then, the second image acquisition unit 14 detects the movement of the user's hand based on the plurality of second feature points 236. Thus, the control device 1 is able to detect the movement of the user's hand, thereby improving the accuracy of detecting the user's hand movement.

[0082] return Figure 5 Continuing the explanation, the storage unit 30 stores various types of information. The storage unit 30 is implemented using hardware such as a memory or storage device for storing information (in other words, data). Specifically, the storage unit 30 stores first coordinate information 31, first image information 32, second image information 33, projective transformation image information 34, third image information 35, optical axis transformation image information 36, first feature point information 37, fourth image information 38, and second feature point information 39.

[0083] The first coordinate information 31 includes the three-dimensional coordinates of the setting position of the sensor 500 (which has the first sensor 501 and the second sensor 502), the installation angle of the sensor 500, and the three-dimensional coordinates of the position of the operating device 600. The first image information 32 includes an image of the user's body captured by the first sensor 501 and the date and time information when the first sensor 501 took the image. The second image information 33 includes an image of the user's wrist captured by the second sensor 502 and the date and time information when the second sensor 502 took the image.

[0084] Projective transformation image information 34 includes information about the projective transformation image obtained by the control device 1 projectively transforming the first image in a manner corresponding to the coordinate system of the second camera. Third image information 35 includes a third image generated by the control device 1. The third image is generated by combining the first and second images and includes the user's body and wrist. Optical axis transformation image information 36 includes information about the optical axis transformation image obtained by the control device 1 projecting the second image in a manner consistent with the optical axis of the first sensor 501.

[0085] The first feature point information 37 includes information on multiple first feature points corresponding to the user's body, determined by the control device 1 from the third image. The first feature points include multiple third feature points corresponding to the wrist. The fourth image information 38 includes information on a fourth image acquired by the control device 1, including the user's hand. The second feature point information 39 includes information on multiple second feature points 236 corresponding to the user's hand, detected by the control device 1 from the fourth image.

[0086] Next, use Figure 10 The processing procedure of the control device 1 involved in the first embodiment will be explained. Figure 10 This is a flowchart illustrating the processing procedure of the control device 1 according to the first embodiment. Furthermore, the process for detecting the user's hand will be described.

[0087] First, the first image acquisition unit 11 acquires a first image including the user's body captured by the first sensor 501 and a second image including the user's wrist captured by the second sensor 502 (step S101). Next, the image transformation unit 12 transforms the first image acquired by the first image acquisition unit 11 into a projective transformation image obtained by performing a projective transformation in a manner corresponding to the coordinate system of the second camera (step S102).

[0088] Next, the image synthesis unit 13 generates a third image including the user's body and wrist by combining the first image acquired by the first image acquisition unit 11 with the second image (step S103). For example, the image synthesis unit 13 generates a third image including the user's body and wrist by combining the projective transformed image obtained by the image transformation unit 12 performing a projective transformation on the first image with an optical axis transformed image obtained by performing an optical axis transformation on the second image in a manner consistent with the optical axis of the first sensor 501.

[0089] Next, the second image acquisition unit 14 determines a plurality of first feature points corresponding to the user's body from the third image including the user's body and wrist (step S104). Then, the second image acquisition unit 14 acquires a fourth image including the user's hand based on a plurality of third feature points corresponding to the wrist included among the plurality of first feature points (step S105). Next, the detection unit 15 determines a plurality of second feature points corresponding to the user's hand from the fourth image including the user's hand, and detects the movement of the user's hand based on the plurality of second feature points (step S106). When the processing of step S106 ends, the control device 1 performs, for example, processing to control the operation device 600 based on the movement of the user's hand.

[0090] As described above, in one aspect of this disclosure, the control device 1 acquires a first image including the user's body captured by a first camera, and a second image including the user's wrist captured by a second camera. Furthermore, the control device 1 generates a third image including the body and wrist by combining the first and second images. Then, the control device 1 determines multiple first feature points corresponding to the body from the third image, and acquires a fourth image including the user's hand based on multiple third feature points corresponding to the wrist. Finally, the control device 1 determines multiple second feature points corresponding to the hand from the fourth image, and detects hand movement based on these multiple second feature points.

[0091] In addition, the control device 1 transforms the first image into a projective transformation image obtained by projecting the first image in a manner corresponding to the coordinate system of the second camera, and combines the projective transformation image with an optical axis transformation image obtained by transforming the second image in a manner consistent with the optical axis of the first camera, thereby generating a third image.

[0092] For example, sometimes a camera mounted inside a vehicle may not capture both the user's body and hands within its field of view, resulting in an image that does not include the user's body or hands. Even in such cases, the control device 1 determines multiple first feature points corresponding to the user's body from a composite image that includes both the image of the user's body and the image of the user's wrist. The control device 1 then acquires an image including the user's hand based on a third feature point corresponding to the wrist, which is included among the determined multiple first feature points. Next, it determines multiple second feature points corresponding to the user's hand from the image including the user's hand. Thus, the control device 1 can detect the movement of the user's hand based on the determined multiple second feature points. Consequently, the control device 1 can improve the accuracy of detecting the user's hand movement.

[0093] Furthermore, the above-described embodiments can also be appropriately modified and implemented by changing a portion of the structure or function of each of the aforementioned devices. Therefore, several variations related to the above-described embodiments will be described below as other embodiments. Furthermore, the following description will focus primarily on points that differ from the above-described embodiments, and detailed explanations of points common to previously described content will be omitted.

[0094] (Second Implementation)

[0095] In the above embodiment, the control device 1 determines a plurality of first feature points corresponding to the user's body from a third image including the user's body and wrist, and acquires a fourth image including the user's hand based on the plurality of third feature points corresponding to the wrist. However, sometimes the first feature points of the user's body cannot be determined in the third image including the user's body and wrist generated by the image synthesis unit 13. This phenomenon occurs, for example, when a narrow-angle monocular camera is used as the first sensor 501 mounted in the vehicle interior.

[0096] Therefore, in the second embodiment, the following method will be described: if the control device 1 is unable to determine multiple first feature points corresponding to the user's body from the third image including the user's body and wrist, the wrist is determined from the second image including the user's wrist, and a fourth image including the user's hand is obtained.

[0097] Figure 11This diagram illustrates the general structure of an information system including the control device 1, which is the information processing apparatus according to the second embodiment. The information processing apparatus of the second embodiment further implements the first determination unit 16 of the control unit 10 of the control device 1, compared to the information processing apparatus of the first embodiment. The CPU may also use RAM as a working area to execute a program stored in ROM, thereby implementing the first determination unit 16. It can also be said that the control device 1 includes the first determination unit 16. The first determination unit 16 may also be implemented using hardware different from the first image acquisition unit 11, the image transformation unit 12, the image synthesis unit 13, the second image acquisition unit 14, and the detection unit 15.

[0098] The first determination unit 16 determines whether it is possible to determine multiple first feature points corresponding to the body from the third image including the user's body and wrist generated by the image synthesis unit 13.

[0099] When the first determination unit 16 determines that multiple first feature points corresponding to the body can be determined from the third image, the second image acquisition unit 14 determines the multiple first feature points corresponding to the body from the third image, and acquires a fourth image including the hand based on multiple third feature points corresponding to the wrist. For example, the second image acquisition unit 14 determines the third feature points corresponding to the wrist among the multiple first feature points and acquires a fourth image including the hand. Then, the detection unit 15 determines multiple second feature points corresponding to the user's hand from the fourth image including the user's hand, and detects the movement of the user's hand based on the multiple second feature points.

[0100] On the other hand, if the first determination unit 16 determines that it is impossible to determine multiple first feature points corresponding to the body from the third image, the second image acquisition unit 14 fixes the object area of ​​the second image including the user's wrist acquired by the first image acquisition unit 11, and acquires a fourth image including the user's hand. Then, the first determination unit 16 determines whether it is possible to determine multiple third feature points corresponding to the wrist from the fourth image including the hand acquired by the second image acquisition unit 14.

[0101] If the first determination unit 16 determines that it is possible to determine multiple third feature points corresponding to the wrist from the fourth image including the hand, the detection unit 15 determines multiple second feature points corresponding to the user's hand from the fourth image and detects the movement of the user's hand based on the multiple second feature points.

[0102] Furthermore, if the first determination unit 16 determines that it is impossible to determine multiple third feature points corresponding to the wrist from the fourth image including the hand, the second image acquisition unit 14 slides the object region of the second image including the user's wrist acquired by the first image acquisition unit 11. Then, the second image acquisition unit 14 fixes the object region of the second image to acquire the fourth image.

[0103] Then, the second image acquisition unit 14 slides the object area of ​​the second image including the user's wrist, fixes the object area of ​​the second image after sliding, and acquires the fourth image until the first determination unit 16 can determine a plurality of third feature points corresponding to the wrist from the fourth image including the hand.

[0104] Here, use Figure 12 The process by which the second image acquisition unit 14 determines the wrist from a second image including the user's wrist will be described. Figure 12 This is a schematic diagram illustrating the processing of the control device 1 according to the second embodiment. Figure 12 The second image 241 is shown. The second image acquisition unit 14 is along... Figure 12 The arrows shown are processed to determine the wrist from the second image. Additionally, in Figure 12 The image shows object regions 242, 243, 244, and 245 corresponding to the wrist.

[0105] For example, the second image acquisition unit 14, as Figure 12 As shown, the object region 243 is slid from the object region 242 located at the upper left end of the second image 241 toward the right end. When the sliding is completed at the upper right end, it moves toward the object region 244 located at the left end, and then the object region 245 is slid toward the right end again. Then, the second image acquisition unit 14 continues this process sequentially until the sliding is completed at the lower right end, and then returns to the upper left end. Thus, the second image acquisition unit 14 can determine the wrist from the second image including the user's wrist, and acquire a fourth image including the user's hand.

[0106] Next, use Figure 13 The processing procedure of the control device 1 involved in the second embodiment will be explained. Figure 13 This is a flowchart illustrating the processing procedure of the control device 1 according to the second embodiment. Furthermore, Figure 13 The processing of steps S101 to S106 in the flowchart shown is related to... Figure 10 The processes for steps S101 to S106 in the flowchart shown are the same, so the description is omitted.

[0107] In step S201, the first determination unit 16 determines whether it is possible to determine multiple first feature points corresponding to the body from the third image generated by the image synthesis unit 13, which includes the user's body and wrist (step S201). Here, when the first determination unit 16 determines that it is possible to determine multiple first feature points corresponding to the body from the third image (step S201: "Yes"), it proceeds to step S105.

[0108] On the other hand, when the first determination unit 16 determines that it is impossible to determine multiple first feature points corresponding to the body from the third image (step S201: "No"), it proceeds to step S202. In step S202, the second image acquisition unit 14 fixes the object area of ​​the second image including the user's wrist acquired by the first image acquisition unit 11 to acquire a fourth image including the user's hand (step S202). Next, the first determination unit 16 determines whether it is possible to determine multiple second feature points corresponding to the hand from the fourth image including the hand (step S203).

[0109] Here, when the first determination unit 16 determines that multiple second feature points can be determined from the fourth image (step S203: "Yes"), it proceeds to step S106. On the other hand, when the first determination unit 16 determines that multiple second feature points cannot be determined from the fourth image (step S203: "No"), it proceeds to step S204. In step S204, the second image acquisition unit 14 slides the object area of ​​the second image including the user's wrist acquired by the first image acquisition unit 11 (step S204). When step S204 ends, the processing of the control device 1 proceeds to step S202, and subsequent processing continues.

[0110] As explained above, in one embodiment of this disclosure, the control device 1, when unable to determine multiple third feature points corresponding to the user's body from the third image, determines the wrist from the second image including the user's wrist to acquire a fourth image. Thus, even when the feature points of the user's body cannot be determined from the third image including both the user's body and wrist, the control device 1 can determine the wrist from the second image including the user's wrist to acquire a fourth image including the user's hand. Consequently, the control device 1 can improve the accuracy of detecting the movement of the user's hand.

[0111] (First variation)

[0112] In the second embodiment described above, when the control device 1 determines the wrist from the second image including the user's wrist, it performs a sliding process from the upper left to the lower right of the second image, but it is not limited to this. For example, the process by which the control device 1 determines the wrist from the second image including the user's wrist could also be a sliding process from the lower left to the upper right of the second image.

[0113] Figure 14 This is a schematic diagram illustrating the processing of the control device 1 involved in the first modified example. Figure 14 The second image 251 is shown. The second image acquisition unit 14 is along... Figure 14 The arrows shown are processed to determine the wrist from the second image. Additionally, in Figure 14The image shows object regions 252, 253, 254, and 255 corresponding to the wrist.

[0114] For example, the second image acquisition unit 14, as Figure 14 As shown, the object region 253 is slid upwards from the object region 252 located at the lower left end of the second image 251. When the sliding is completed at the upper left end, it moves to the object region 254 located at the lower end, and then the object region 255 is slid upwards again. Then, the second image acquisition unit 14 continues this process sequentially until the sliding is completed at the upper right end, and then returns to the lower left end. As a result, the control device 1 can determine the wrist from the second image including the user's wrist and acquire a fourth image including the user's hand. Therefore, the control device 1 can improve the accuracy of detecting the movement of the user's hand.

[0115] (Second variation)

[0116] In the second embodiment and the first variation described above, the control device 1 determines the wrist from a second image including the user's wrist. In the second variation, the processing in the following situation will be explained: when the control device 1 determines the wrist from the second image including the user's wrist, acquires a fourth image including the hand, and detects hand movement, the detected hand is no longer visible.

[0117] Figure 15 This is a schematic diagram illustrating the processing of the control device 1 involved in the second variation. Figure 15 The second image 261 is shown. The second image acquisition unit 14 is along... Figure 15 The arrows shown are processed to determine the wrist from the second image. Additionally, in Figure 15 The image shows object region 262 and object region 264 corresponding to the wrist. Here, the second image acquisition unit 14 is in a state where the wrist has been identified in object region 262 and a fourth image 263 has been acquired.

[0118] For example, if the hand disappears after the detection unit 15 has detected hand movement in the fourth image 263, the second image acquisition unit 14 needs to acquire the fourth image including the hand again. Therefore, for example, the second image acquisition unit 14... Figure 15 As shown, the control device 1 slides from the object region 264 located at the upper left end of the second image 261 toward the upper right end, and continues processing until a fourth image including the user's hand can be acquired. Therefore, even if the hand movement is lost, the control device 1 can detect hand movement by acquiring a fourth image including the hand again. Thus, the control device 1 can improve the accuracy of detecting the user's hand movement.

[0119] (Third variation)

[0120] Figure 16 This is a schematic diagram illustrating the processing of the control device 1 involved in the third variation. Figure 16 The second image 271 is shown. The second image acquisition unit 14 is along... Figure 16 The arrows shown are processed to determine the wrist from the second image. Additionally, in Figure 16 The image shows object region 272 and object region 274 corresponding to the wrist. Here, the second image acquisition unit 14 is in a state where the wrist has been identified in object region 272 and a fourth image 273 has been acquired.

[0121] For example, if the hand disappears while the detection unit 15 has detected hand movement in the fourth image 273, the second image acquisition unit 14 will... Figure 16 As shown, the object region 272 is slid to the left, that is, slid to the state of object region 274 immediately preceding object region 272, and processing continues until a fourth image including the user's hand can be acquired. Thus, even if hand movement is lost, the control device 1 can detect hand movement by acquiring a fourth image including the hand again. Therefore, the control device 1 can improve the accuracy of detecting the user's hand movement.

[0122] (Third Implementation)

[0123] In the above-described embodiment, when the control device 1 determines the wrist from a second image including the user's wrist, it determines the wrist in all regions of the second image, but is not limited thereto. For example, the control device 1 may also determine a third feature point corresponding to the wrist from a predetermined region in the second image to obtain a fourth image.

[0124] Figure 17 This is a schematic diagram illustrating the processing of the control device 1 according to the third embodiment. Figure 17 The second image 281 is shown. As shown in the second image 281, the user's wrist exists only in a portion of the area in the second image 281. Therefore, the second image acquisition unit 14 determines a third feature point corresponding to the wrist from a predetermined area in the second image 281 to acquire a fourth image.

[0125] For example, such as Figure 17As shown, the second image acquisition unit 14 slides from the object area 282 located at the upper left end of the second image 281 towards the upper right end and then towards the object area 283 in region R1 of the upper half of the second image 281, and continues processing until a fourth image including the user's hand can be acquired. Furthermore, the predetermined set area is not limited to the upper half of the second image 281, and can be set to match the range of motion of the user's hand. Therefore, the control device 1 can shorten the processing time until a fourth image including the user's hand is acquired.

[0126] (Fourth Implementation)

[0127] For example, control device 1 may sometimes be unable to determine multiple second feature points corresponding to the hand because the fourth image, including the user's hand, is unclear. Therefore, the handling of the case where the fourth image including the user's hand is unclear will be explained.

[0128] Figure 18 This diagram illustrates the general structure of an information system including the control device 1, which is an information processing apparatus according to the fourth embodiment. The information processing apparatus of the fourth embodiment, compared to the information processing apparatus of the first embodiment, further implements the second determination unit 17 and the correction unit 18 of the control unit 10 of the control device 1, as well as the brightness value information 40 of the storage unit 30 of the control device 1. Alternatively, the CPU may use RAM as a working area to execute a program stored in ROM, thereby implementing the correction unit 18. It may also be said that the control device 1 includes the correction unit 18. The correction unit 18 may also be implemented using hardware different from the first image acquisition unit 11, the image transformation unit 12, the image synthesis unit 13, the second image acquisition unit 14, and the detection unit 15.

[0129] The second determination unit 17 determines whether the brightness value of the fourth image, including the user's hand, acquired by the second image acquisition unit 14 meets a predetermined condition. If the second determination unit 17 determines that the brightness value of the fourth image meets the predetermined condition, the detection unit 15 determines a plurality of second feature points corresponding to the user's hand from the fourth image, and detects the movement of the user's hand based on the plurality of second feature points.

[0130] If the second determination unit 17 determines that the brightness value of the fourth image including the user's hand does not meet the specified conditions, the correction unit 18 corrects the brightness value of the fourth image.

[0131] Here, the correction process performed by the correction unit 18 will be explained. Figure 19 This is a schematic diagram illustrating the processing of the control device 1 according to the fourth embodiment. Figure 19 The fourth image 291 before correction and the fourth image 292 after correction are shown.

[0132] For example, if the brightness value of the fourth image 291 does not meet a predetermined condition, the correction unit 18 generates a fourth image 292 by correcting the brightness value of the fourth image. Here, the correction unit 18, for example, adds the brightness of each pixel of the fourth image 291, and if the total value obtained is above a predetermined brightness value, it corrects by decreasing the brightness value. Conversely, if the total value is below a predetermined brightness value, the correction unit 18 corrects by increasing the brightness value. The brightness value correction process is, for example, the known dynamic range expansion process. Therefore, the control device 1 can make the fourth image, including the user's hand, clearer.

[0133] return Figure 18 Continuing the explanation, the brightness value information 40 includes the brightness values ​​specified above. The conditions for specifying the brightness values ​​are set according to the specifications of the sensor 500.

[0134] Next, use Figure 20 The processing procedure of the control device 1 involved in the fourth embodiment will be explained. Figure 20 This is a flowchart illustrating the processing procedure of the control device 1 according to the fourth embodiment. Furthermore, Figure 20 The processing of steps S101 to S106 in the flowchart shown is related to... Figure 10 The processes for steps S101 to S106 in the flowchart shown are the same, so the description is omitted.

[0135] In step S301, the second determination unit 17 determines whether the brightness value of the fourth image, including the user's hand, acquired by the second image acquisition unit 14 meets the predetermined conditions (step S301). Here, when the second determination unit 17 determines that the brightness value of the fourth image meets the predetermined conditions (step S301: "Yes"), it proceeds to step S106.

[0136] On the other hand, when the second determination unit 17 determines that the brightness value of the fourth image does not meet the prescribed conditions (step S301: "No"), it proceeds to step S302. In step S302, the correction unit 18 corrects the brightness value of the fourth image (step S302). When step S302 ends, the processing of the control device 1 proceeds to step S106.

[0137] As explained above, in one aspect of this disclosure, the control device 1 corrects the brightness value of the fourth image when the brightness value of the fourth image does not meet the specified conditions. As a result, the fourth image, including the user's hand, becomes clear, and the control device 1 is able to detect the movement of the user's hand, thus improving the accuracy of detecting the user's hand movement.

[0138] (Fourth variation)

[0139] As a fourth variation, an information processing device with a single sensor 500 will be described. For example, in the above embodiment, images from two cameras are combined. In contrast, in the fourth variation, a still image including the user's body is combined with a camera image showing the user's wrist and hand. Thus, even with only one camera, the movement of the user's hand can be detected. Furthermore, in the second and third embodiments described above, as well as the first, second, and third variations, a third image including the user's body and wrist is generated by combining images captured using the first sensor 501 and the second sensor 502. If multiple first feature points corresponding to the user's body cannot be determined from the third image, a new object region is determined by sliding an object region in the second image including the user's wrist. Then, a fourth image including the user's hand is acquired, and second feature points corresponding to the hand are determined to detect the movement of the user's hand.

[0140] In contrast, the fourth modification involves a single sensor 500. Therefore, depending on the location of the sensor 500 mounted inside the vehicle interior, sometimes it may be possible to capture the user's wrist and hand, but not the body. In this case, the information processing apparatus of the fourth modification determines the wrist from an image captured by a camera that includes at least the user's wrist, and acquires an image including the user's hand based on the determined wrist. Then, hand movement is detected from the image including the user's hand. Similarly, the fourth embodiment described above can be applied to the information processing apparatus of the fourth modification.

[0141] For example, the information processing apparatus involved in the fourth modification may also be an information processing apparatus that performs actions based on the movement of a user's hand. This information processing apparatus includes: a first image acquisition unit 11 that acquires an image captured by a camera that includes at least the user's wrist; a second image acquisition unit 14 that determines a plurality of third feature points corresponding to the wrist from the image including the wrist, and acquires an image including the user's hand based on the plurality of third feature points; and a detection unit 15 that determines a plurality of second feature points corresponding to the user's hand from the image including the hand, and detects hand movement based on the plurality of second feature points.

[0142] Therefore, control device 1 can determine the wrist from an image including the user's wrist, and thus acquire an image including the user's hand. Therefore, even with only one sensor 500, control device 1 can improve the accuracy of detecting the user's hand movements.

[0143] (Fifth Implementation)

[0144] For example, sensor 500, in addition to the above... Figure 3 In addition to the configuration state D2 shown, the sensor 500 can also be mounted inside the vehicle interior. Therefore, the configuration of the sensor 500 inside the vehicle interior will be described.

[0145] Figure 21 This is a schematic diagram illustrating the configuration of the sensor 500 according to the fifth embodiment. Figure 21 The schematic diagram shows the sensor 500 mounted on the vehicle, capable of capturing images. Figure 2 The configuration state D3 is shown in the position of image 201. The sensor 500 according to the fifth embodiment is configured in region R2 under configuration state D3. Region R2 is the area located in front of and in front of the user inside the vehicle.

[0146] Figure 22 This is a schematic diagram illustrating the configuration of the sensor 500 according to the fifth embodiment. Figure 22 The diagram shown illustrates the application of... Figure 21 The region R2 under configuration state D3 shown is configured with the states of the first sensor 501 and the second sensor 502.

[0147] The first sensor 501 is configured, for example, in the YZ plane facing the positive direction of the Z-axis to capture images including the user's face and body. The second sensor 502 is configured, for example, in the YZ plane facing the negative direction of the Y-axis to capture images including the user's arms, wrists, and hands.

[0148] Figure 23 This is a schematic diagram illustrating the processing of the control device 1 according to the fifth embodiment. Figure 23 The processing performed by the image synthesis unit 13 of the control unit 10 of the control device 1 will be explained. Figure 23 The image 303 generated by the image compositing unit 13 is shown. For example, Figure 23 The third image 303 shown is... Figure 22 The image 301 captured by the first sensor 501 shown is... Figure 22 The image is synthesized from the image captured by the second sensor 502 shown in Figure 302.

[0149] Furthermore, images 301 and 302 are images captured with the optical axis of the first sensor 501 intersecting the optical axis of the second sensor 502. Therefore, the image synthesis unit 13 can generate an image including the user's body and wrist by combining the images acquired from the first sensor 501 and the second sensor 502 mounted in the vehicle interior.

[0150] (Fifth variation)

[0151] In the fifth variation, the configuration of the sensor 500 in the vehicle interior, which differs from that in the fifth embodiment, will be described. Figure 24 This is a schematic diagram illustrating the configuration of the sensor 500 involved in the fifth variation.Figure 24 The schematic diagram shown

[0152] Showing targets Figure 21 The area R2 under configuration state D3 shown is configured with the first sensor 501 and the second sensor 502.

[0153] The first sensor 501 is configured, for example, in the YZ plane facing the negative direction of the Y-axis and the positive direction of the Z-axis, to capture images including the user's face.

[0154] The second sensor 502 is configured, for example, in the YZ plane facing the negative direction of the Y-axis and the negative direction of the Z-axis, to capture images including the user's body, arms, wrists, and hands.

[0155] Figure 25 This is a schematic diagram illustrating the processing of the control device 1 involved in the fifth modification. Figure 25 The processing performed by the image synthesis unit 13 of the control unit 10 of the control device 1 will be explained.

[0156] Figure 25 The image 313 generated by the image compositing unit 13 is shown. For example, Figure 25 The third image 313 shown is... Figure 24 The image 311 captured by the first sensor 501 shown is... Figure 24 The image is synthesized from image 312 captured by the second sensor 502 shown. Furthermore, images 311 and 312 are images captured with the optical axis of the first sensor 501 intersecting the optical axis of the second sensor 502. Therefore, the image synthesis unit 13 can generate an image including the user's body and wrist by synthesizing the images acquired from the first sensor 501 and the second sensor 502 mounted in the vehicle interior.

[0157] (Sixth variation)

[0158] In the sixth variation, the configuration of the sensor 500 in the vehicle interior, which differs from that in the fifth embodiment and the fifth variation, will be described. Figure 26 This is a schematic diagram illustrating the configuration of the sensor 500 involved in the sixth variation. Figure 26 The diagram shown illustrates the application of... Figure 21 The region R2 under configuration state D3 shown is configured with the first sensor 501 and the second sensor 502. The first sensor 501 is configured, for example, in the YZ plane facing the negative direction of the Y-axis and the positive direction of the Z-axis, to capture images including the user's face, body, and arms.

[0159] The second sensor 502 is configured, for example, in the YZ plane facing the negative direction of the Y-axis and the negative direction of the Z-axis, to capture images including the user's wrist and hand. Figure 27 This is a schematic diagram illustrating the processing of the control device 1 involved in the sixth modification. Figure 27 The processing performed by the image synthesis unit 13 of the control unit 10 of the control device 1 will be explained.

[0160] Figure 27 The third image 323 generated by the image compositing unit 13 is shown. For example, Figure 27 The third image 323 shown is... Figure 26 The image 321 captured by the first sensor 501 shown is... Figure 26 The image is synthesized from the image 322 captured by the second sensor 502 shown. Furthermore, images 321 and 322 are images captured in a manner where the optical axes of the first sensor 501 and the second sensor 502 do not intersect. Therefore, the image synthesis unit 13 can generate an image including the user's body and wrist by synthesizing the individual images acquired from the first sensor 501 and the second sensor 502 mounted in the vehicle interior.

[0161] (Sixth Implementation Method)

[0162] In the first embodiment described above, the control device 1 generates a third image as a composite image including the user's body and wrist, determines a plurality of first feature points corresponding to the body from the third image, and uses a second image corresponding to a plurality of third feature points corresponding to the wrist to obtain a fourth image including the user's hand. In contrast, in the sixth embodiment, instead of generating a composite image including the user's body and wrist (i.e., the third image), the fourth image including the user's hand is obtained using the second image including the user's wrist.

[0163] Figure 28 This diagram illustrates a general structure of an information system including a control device 1, which is part of the information processing apparatus according to the sixth embodiment. The information processing apparatus according to the sixth embodiment includes a sensor 500, an operating device 600, and a control device 1. The control device 1 includes a control unit 10 and a storage unit 30. The control unit 10 includes a first image acquisition unit 11, a second image acquisition unit 14, and a detection unit 15. Furthermore, descriptions of contents identical to those in the first embodiment are omitted.

[0164] For example, sometimes the image captured by the first sensor 501 includes the user's body and wrist. In this case, the control device 1 can determine the user's wrist from the image including the user's body and wrist. The processing of the specific functional unit will be described below.

[0165] The first image acquisition unit 11 acquires a fifth image captured by the first sensor 501, including the user's body and wrist, and a second image captured by the second camera, including the user's wrist.

[0166] The second image acquisition unit 14 determines multiple third feature points corresponding to the wrist from the fifth image, transforms the multiple third feature points into coordinates of the second image, and acquires a fourth image including the user's hand based on the multiple third feature points in the second image. Here, using Figure 29 Let's explain the coordinate transformation image.

[0167] Figure 29 This is a schematic diagram illustrating the processing of the control device 1 according to the sixth embodiment. Figure 29 The image shows a fifth image 331 captured by the first sensor 501, including the user's body and wrist, and coordinates 332 of the four corners when the hand moves forward left and forward right, and backward left and backward right. It also shows a second image 333 captured by the second camera, including the user's wrist, and coordinates 334 of the four corners when the hand moves forward left and forward right, and backward left and backward right.

[0168] Here, the fifth image 331 includes the user's body and wrist, but only a portion of the user's hand. In other words, the fifth image 331 does not include the user's entire hand. Therefore, it is impossible to determine the multiple second feature points corresponding to the user's hand solely from the fifth image 331. On the other hand, the second image 333 includes the user's wrist and hand.

[0169] First, the second image acquisition unit 14 determines a plurality of third feature points corresponding to the wrist from the fifth image 331 acquired by the first image acquisition unit 11. Furthermore, the second image acquisition unit 14 transforms the coordinates of the plurality of third feature points in the fifth image 331 to the coordinates of the second image 333 by using a projective transformation of the coordinate points 332 of the fifth image 331 and the coordinate points 334 of the second image 333. Then, the second image acquisition unit 14 determines the plurality of third feature points in the second image 333 and acquires a fourth image including the user's hand based on the plurality of third feature points in the second image 333.

[0170] Therefore, when the image acquired by the first image acquisition unit 11 includes the user's body and wrist, the second image acquisition unit 14 can acquire an image including the user's hand from the image captured by the second sensor 502 without combining the image captured by the first sensor 501 with the image captured by the second sensor 502.

[0171] return Figure 28Continuing the explanation. The storage unit 30 stores the first coordinate information 31, the second image information 33, the first feature point information 37, the fourth image information 38, the second feature point information 39, the fifth image information 41, and the coordinate transformation image information 42.

[0172] The fifth image information 41 includes an image of the user's body and wrist captured by the first sensor 501, as well as the date and time when the first sensor 501 captured the image. The coordinate transformation image information 42 includes information about the coordinate transformation image obtained by the second image acquisition unit 14 projectively transforming the fifth image in a manner corresponding to the coordinate system of the second camera.

[0173] Next, use Figure 30 The processing procedure of the control device 1 involved in the sixth embodiment will be explained. Figure 30 This is a flowchart illustrating the processing procedure of the control device 1 according to the sixth embodiment. Furthermore, Figure 30 The processing of step S105 in the flowchart shown is related to... Figure 10 The process of step S105 in the flowchart shown is the same, so the description is omitted.

[0174] In step S401, the first image acquisition unit 11 acquires a fifth image including the user's body and wrist captured by the first sensor 501, and a second image including the user's wrist captured by the second sensor 502 (step S401). In step S402, the second image acquisition unit 14 determines a plurality of third feature points corresponding to the wrist from the fifth image 331 acquired by the first image acquisition unit 11 (step S402).

[0175] In step S403, the second image acquisition unit 14 transforms the coordinates of multiple third feature points of the fifth image 331 to the coordinates of the second image 333 by using the projective transformation of coordinate points 332 of the fifth image 331 and coordinate points 334 of the second image 333 (step S403). In step S404, the second image acquisition unit 14 determines multiple third feature points in the second image 333 and acquires a fourth image including the user's hand based on the multiple third feature points in the second image 333 (step S404). When step S404 ends, the processing of the control device 1 proceeds to step S106.

[0176] As explained above, in one aspect of this disclosure, the control device 1 acquires a fifth image captured by a first camera, including the user's body and wrist, and a second image captured by a second camera, including the user's wrist. Furthermore, the control device 1 determines multiple third feature points corresponding to the wrist from a coordinate-transformed image obtained by projectively transforming the fifth image in a manner corresponding to the coordinate system of the second camera. Using the second image corresponding to these multiple third feature points, it acquires a fourth image including the user's hand. Then, the control device 1 determines multiple second feature points corresponding to the hand from the fourth image and detects hand movement based on these multiple second feature points.

[0177] For example, even when the acquired image includes only the user's body and not the hand, the control device 1 determines multiple third feature points corresponding to the wrist from the image including the user's body and wrist. Based on the determined multiple third feature points, it acquires an image including the user's hand and determines multiple second feature points corresponding to the user's hand from the image including the user's hand. Therefore, the control device 1 can detect the movement of the user's hand based on the determined multiple second feature points. Consequently, the control device 1 can improve the accuracy of detecting the user's hand movement.

[0178] The embodiments of this disclosure have been described above, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These new embodiments and their variations are included within the scope and spirit of the invention, and are included within the scope of the invention as described in the claims and its equivalents. Furthermore, the constituent elements of different embodiments and variations can be appropriately combined.

[0179] In addition, the term "...part" in the above embodiments can be replaced with other terms such as "...circuitry", "...component", "...device", "...unit" or "...module".

[0180] In the above embodiments, examples of hardware configurations have been described, but the present disclosure can also be implemented in software in cooperation with hardware.

[0181] Furthermore, the functional blocks used in the descriptions of the above embodiments are typically implemented as integrated circuits, i.e., LSIs. Integrated circuits can also control the functional blocks used in the descriptions of the above embodiments, and have input terminals and output terminals. They can be implemented on a single chip individually, or in a manner that includes some or all of them. Here, it is referred to as an LSI, but depending on the level of integration, it is sometimes also called an IC, system-on-a-large-scale integrated circuit (system LSI), super-large-scale integrated circuit (super LSI), or ultra-ultra-large-scale integrated circuit (ultra LSI).

[0182] Furthermore, the integration method is not limited to LSI; it can also be implemented using dedicated circuits or general-purpose processors and memory. After LSI manufacturing, programmable FPGAs (Field Programmable Gate Arrays) and reconfigurable processors that can reconfigure the connections or settings of the internal circuit cells of the LSI can also be used.

[0183] Furthermore, if advancements in semiconductor technology or other derived technologies lead to the development of integrated circuit technologies that replace LSIs, then these technologies can certainly be used to integrate functional blocks. Applications in biotechnology, among others, are also possible.

[0184] Furthermore, the effects described in the embodiments in this specification are merely illustrative and not limiting; other effects may also be achieved.

[0185] (Postscript)

[0186] The following technology has been disclosed through the above description of the embodiments.

[0187] (Technology 1)

[0188] An information processing device comprising:

[0189] The first image acquisition unit acquires a first image of the user's body captured by a first camera, and a second image of the user's wrist captured by a second camera;

[0190] An image synthesis unit generates a third image including the body and the wrist by compositing the first image with the second image;

[0191] The second image acquisition unit determines multiple first feature points corresponding to the body from the third image, and acquires a fourth image including the user's hand based on multiple third feature points corresponding to the wrist; and

[0192] The detection unit determines a plurality of second feature points corresponding to the hand from the fourth image, and detects the movement of the hand based on the plurality of second feature points.

[0193] (Technology 2)

[0194] According to the information processing apparatus of the aforementioned technology 1, wherein...

[0195] It also includes an image transformation unit that transforms the first image into a projective transformed image obtained by performing a projective transformation in a manner corresponding to the coordinate system of the second camera.

[0196] The image compositing unit generates the third image by compositing the projective transformation image and the optical axis transformation image. The optical axis transformation image is obtained by transforming the second image in a manner consistent with the optical axis of the first camera.

[0197] (Technology 3)

[0198] According to the information processing apparatus of the above-described technology 2, wherein...

[0199] If the second image acquisition unit cannot determine the plurality of first feature points from the third image, it determines a third feature point corresponding to the wrist from the second image and acquires the fourth image.

[0200] (Technology 4)

[0201] According to the information processing apparatus described in technique 3 above, wherein...

[0202] The second image acquisition unit determines a third feature point corresponding to the wrist from a pre-defined area of ​​the second image and acquires the fourth image.

[0203] (Technology 5)

[0204] An information processing device comprising:

[0205] The first image acquisition unit acquires a fifth image captured by the first camera, including the user's body and wrist, and a second image captured by the second camera, including the user's wrist.

[0206] The second image acquisition unit determines a plurality of third feature points corresponding to the wrist from a coordinate-transformed image obtained by projectively transforming the fifth image to the coordinate system of the second camera, and uses the second image corresponding to the plurality of third feature points to acquire a fourth image including the user's hand; and

[0207] The detection unit determines a plurality of second feature points corresponding to the hand from the fourth image, and detects the movement of the hand based on the plurality of second feature points.

[0208] (Technology 6)

[0209] The information processing apparatus according to any one of the above-described technologies 1 to 5 further includes a correction unit that corrects the brightness value of the fourth image when the brightness value of the fourth image does not meet a predetermined condition.

[0210] (Technology 7)

[0211] An information processing device comprising:

[0212] The image acquisition unit acquires an image of the user's wrist captured by the camera;

[0213] The second image acquisition unit determines a plurality of third feature points corresponding to the wrist from an image including the wrist, and acquires an image including the user's hand based on the plurality of third feature points; and

[0214] The detection unit determines a plurality of second feature points corresponding to the user's hand from an image including the hand, and detects the movement of the hand based on the plurality of second feature points.

[0215] (Technology 8)

[0216] According to the information processing apparatus described in technique 7 above, wherein...

[0217] The second image acquisition unit determines the wrist from a pre-defined region including the image of the wrist, and acquires an image including the hand.

[0218] (Technology 9)

[0219] According to the information processing apparatus described in technique 7 or 8 above, wherein,

[0220] It also includes a correction unit that corrects the brightness value of the image including the hand when the brightness value does not meet the specified conditions.

[0221] (Technology 10)

[0222] An information processing system, comprising:

[0223] Multiple cameras, including a first camera and a second camera; and

[0224] The information processing device according to the above-described technology 1 is connected to the plurality of cameras.

[0225] (Technology 11)

[0226] An information processing method includes the following steps:

[0227] The first image acquisition step involves acquiring a first image captured by a first camera, including the user's body, and a second image captured by a second camera, including the user's wrist.

[0228] The image compositing step generates a third image including the body and the wrist by compositing the first image with the second image;

[0229] The second image acquisition step involves determining multiple first feature points corresponding to the body from the third image, and acquiring a fourth image including the user's hand based on multiple third feature points corresponding to the wrist; and

[0230] The detection step involves determining multiple second feature points corresponding to the hand from the fourth image, and detecting the movement of the hand based on the multiple second feature points.

[0231] (Technology 12)

[0232] A computer program for causing a computer to perform the information processing method according to the above-described technique 11.

Claims

1. An information processing apparatus comprising: a first image acquisition section that acquires a first image including a body of a user, which is captured by a first camera, and a second image including a wrist of the user, which is captured by a second camera; an image synthesis section that generates a third image including the body and the wrist by synthesizing the first image and the second image; a second image acquisition section that determines a plurality of first feature points corresponding to the body from the third image, and acquires a fourth image including a hand of the user based on a third feature point corresponding to the wrist, which is included in the plurality of first feature points; and a detection section that determines a plurality of second feature points corresponding to the hand from the fourth image, and detects a motion of the hand based on the plurality of second feature points. 2.The information processing apparatus according to claim 1, wherein an image transformation section that transforms the first image into a projective transformation image obtained by performing projective transformation in a manner corresponding to a coordinate system of the second camera is further included, the image synthesis section generates the third image by synthesizing the projective transformation image and an optical axis transformation image obtained by performing optical axis transformation on the second image in a manner in which an optical axis of the first camera is coincident. 3.The information processing apparatus according to claim 2, wherein the second image acquisition section determines the third feature point from the second image and acquires the fourth image in a case where the plurality of first feature points cannot be determined from the third image. 4.The information processing apparatus according to claim 3, wherein the second image acquisition section determines the third feature point corresponding to the wrist from within a predetermined region of the second image, and acquires the fourth image. 5.The information processing apparatus according to any one of claims 1 to 4, wherein a correction section that corrects a luminance value of the fourth image in a case where the luminance value of the fourth image does not satisfy a prescribed condition is further included. 6.An information processing apparatus comprising: a first image acquisition section that acquires a fifth image including a body and a wrist of a user, which is captured by a first camera, and a second image including the wrist of the user, which is captured by a second camera; a second image acquisition section that determines a plurality of third feature points corresponding to the wrist from a coordinate transformation image obtained by performing projective transformation on the fifth image to a coordinate system of the second camera, and acquires a fourth image including a hand of the user using the second image corresponding to the plurality of third feature points; and a detection section that determines a plurality of second feature points corresponding to the hand from the fourth image, and detects a motion of the hand based on the plurality of second feature points. 7.The information processing apparatus according to claim 6, wherein a correction section that corrects a luminance value of the fourth image in a case where the luminance value of the fourth image does not satisfy a prescribed condition is further included. 8.An information processing apparatus comprising: ​ an image acquisition section that acquires an image including a wrist of a user captured by a camera; a second image acquisition section that determines a plurality of third feature points corresponding to the wrist from the image including the wrist, and acquires an image including a hand of the user based on the plurality of third feature points; and a detection section that determines a plurality of second feature points corresponding to the hand from the image including the hand, and detects a motion of the hand based on the plurality of second feature points.

9. The information processing apparatus according to claim 8, wherein the second image acquisition section determines the wrist from within a preset region of the image including the wrist, and acquires the image including the hand.

10. The information processing apparatus according to claim 8 or 9, wherein further comprising a correction section that corrects a luminance value of the image including the hand in a case where the luminance value does not satisfy a prescribed condition.

11. An information processing system comprising: a camera; and an information processing apparatus connected to the camera, wherein the camera having a first camera and a second camera, the information processing apparatus comprising: a first image acquisition section that acquires a first image including a body of a user captured by the first camera, and a second image including a wrist of the user captured by the second camera; an image synthesis section that generates a third image including the body and the wrist by synthesizing the first image and the second image; a second image acquisition section that determines a plurality of first feature points corresponding to the body from the third image, and acquires a fourth image including a hand of the user based on a plurality of third feature points corresponding to the wrist included in the plurality of first feature points; and a detection section that determines a plurality of second feature points corresponding to the hand from the fourth image, and detects a motion of the hand based on the plurality of second feature points.

12. An information processing method executed by an information processing apparatus, the information processing method comprising: acquiring a first image including a body of a user captured by a first camera, and a second image including a wrist of the user captured by a second camera; generating a third image including the body and the wrist by synthesizing the first image and the second image, determining a plurality of first feature points corresponding to the body from the third image, and acquiring a fourth image including a hand of the user based on a plurality of third feature points corresponding to the wrist included in the plurality of first feature points; and determining a plurality of second feature points corresponding to the hand from the fourth image, and detecting a motion of the hand based on the plurality of second feature points.

13. A recording medium having a program stored therein, the program causing a computer to execute: a first image acquisition step of acquiring a first image including a body of a user captured by a first camera, and a second image including a wrist of the user captured by a second camera; an image synthesis step of generating a third image including the body and the wrist by synthesizing the first image and the second image; a second image acquisition step of acquiring a fourth image including a hand of the user from the third image based on a plurality of third feature points corresponding to the wrist included in the plurality of first feature points; and a detection step of detecting a motion of the hand based on a plurality of second feature points corresponding to the hand determined from the fourth image.

14. A computer program product containing a program which causes a computer to execute the steps of: a first image acquisition step of acquiring a first image including a body of a user taken by a first camera and a second image including a wrist of the user taken by a second camera; an image synthesis step of generating a third image including the body and the wrist by synthesizing the first image with the second image; a second image acquisition step of acquiring a fourth image including a hand of the user from the third image based on a plurality of third feature points corresponding to the wrist included in the plurality of first feature points; and a detection step of detecting a motion of the hand based on a plurality of second feature points corresponding to the hand determined from the fourth image.

Images