Information processing apparatus, information processing system, and information processing method

By acquiring and detecting the user's arm part, setting the virtual operation area and touch determination surface based on the arm position and length, solving the convenience and error detection problems of virtual operation area in the prior art, and realizing user-friendly operation control.

CN119013639APending Publication Date: 2024-11-22PANASONIC AUTOMOTIVE SYST CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202380025591.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-31
Filing Date
2023-01-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the prior art, setting of the virtual operation area is difficult to meet the user's operation convenience, and it is easy to lead to false detection and unintentional operation.

Method used

By acquiring and detecting the user's arm portion, setting the virtual operation area and touch determination surface based on the position and length of the arm, combining 3D and 2D coordinate operations, the user's fingertip movement is determined to realize control of the operating device.

Benefits of technology

It effectively sets a virtual operation area that is easy for users to operate, reduces false detection and unintentional operations, and improves operation convenience and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013639A_ABST
    Figure CN119013639A_ABST
Patent Text Reader

Abstract

In an information processing device of the present disclosure, an image obtained by capturing an image of a user is acquired; detecting an arm portion of the user based on the image; setting a virtual operation region for receiving an operation performed on the device on the basis of the detected position of the arm portion of the user and the length of the arm portion of the user; according to the image, the device is operated based on a positional relationship between the fingertip of the user and the virtual operation area.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method. Background Art

[0002] Conventionally, there are various devices that detect the movement of a user's hand and perform action control according to the movement of the user's hand (for example, see Patent Document 1). For example, for an operation object such as a display, the content displayed on the display is changed according to the movement of the user's hand without the user directly touching the display.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-32055 Summary of the invention

[0006] In the above-mentioned technology, it is considered to set a virtual operation area for specifying a user's selection operation, etc. It is desirable to set such a virtual operation area so that the user can operate it easily.

[0007] The present disclosure provides an information processing device capable of setting a virtual operation area that is easy for a user to operate.

[0008] The information processing device involved in the present disclosure includes: an image acquisition unit, which acquires an image obtained by photographing a user; a detection unit, which detects the user's arm part based on the image; a setting unit, which sets a virtual operation area for accepting operations performed on a device based on the position of the user's arm part and the length of the user's arm part detected by the detection unit; and an operation unit, which operates the device based on the positional relationship between the user's fingertips and the virtual operation area according to the image.

[0009] According to the information processing device according to the present disclosure, it is possible to set a virtual operation area that is easy for the user to operate. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram showing an example of an information processing device.

[0011] Figure 2A 1 is a diagram showing an example of operation instructions viewed from above.

[0012] Figure 2B It is a diagram showing an example of an operation instruction viewed from the side.

[0013] Figure 3A It is a diagram showing an example of the detection range.

[0014] Figure 3B This is a table showing whether each area can be detected.

[0015] Figure 4A This is a diagram showing an example of erroneous detection.

[0016] Figure 4B It is a diagram showing another example of erroneous detection.

[0017] Figure 5 This is a block diagram showing the functional configuration of the information processing device according to the first embodiment.

[0018] Fig. 6A FIG. 1 is a diagram showing a touch determination surface viewed from above.

[0019] Figure 6B FIG. 1 is a diagram showing a touch determination surface viewed from the side.

[0020] Fig. 7A FIG. 1 is a diagram showing a touch determination surface viewed from above.

[0021] Figure 7B FIG. 1 is a diagram showing a touch determination surface viewed from the side.

[0022] Fig. 8A FIG. 1 is a diagram showing a touch determination surface viewed from above.

[0023] Figure 8B FIG. 1 is a diagram showing a touch determination surface viewed from the side.

[0024] Fig.9A FIG. 1 is a diagram showing a touch determination surface viewed from above.

[0025] Fig. 9B FIG. 1 is a diagram showing a touch determination surface viewed from the side.

[0026] Fig. 10A : is a figure which shows the detection effective area viewed from the top.

[0027] Fig. 10B It is a diagram showing the detection effective area viewed from the side.

[0028] Fig.11 It is a flowchart showing the processing procedure involved in the second embodiment.

[0029] Fig.12 It is a diagram showing the effects obtained by the control device according to the second embodiment.

[0030] Fig.13 FIG. 1 is a diagram showing an example of setting a touch determination surface for each of a plurality of users.

[0031] Fig.14It is a flowchart showing the processing procedure involved in the third embodiment.

[0032] Fig.15 It is a diagram for explaining an example of image removal according to the fourth embodiment.

[0033] Fig.16A It is a diagram for explaining an example of image removal according to the fourth embodiment.

[0034] Fig. 16B It is a diagram for explaining an example of image removal according to the fourth embodiment.

[0035] Fig.17 It is a flowchart showing the processing procedure involved in the fourth embodiment.

[0036] Fig.18 It is a diagram showing the arrangement positions of sensors according to a modification example.

[0037] Fig.19 It is a diagram showing the arrangement positions of sensors according to a modification example.

[0038] Fig. 20 It is a diagram showing the arrangement positions of sensors according to a modification example.

[0039] Fig.21 A diagram showing an example of a display according to a modification.

[0040] Fig. 22 FIG. 1 is a diagram showing examples of touch determination surfaces of a plurality of devices.

[0041] Fig.23 FIG. 1 is a diagram showing examples of touch determination surfaces of a plurality of devices.

[0042] Fig.24A FIG. 1 is a diagram showing a touch determination surface based on the left arm.

[0043] Fig. 24B FIG. 1 is a diagram showing a touch determination surface based on the right arm.

[0044] Fig.25A It is a diagram showing the effective detection area by both arms.

[0045] Fig.25B It is a diagram showing the effective detection area by both arms. DETAILED DESCRIPTION

[0046] Hereinafter, embodiments of the information processing device according to the present disclosure will be described with reference to the drawings.

[0047] (First Embodiment)

[0048] Before describing the information processing device according to the first embodiment, a device that operates based on the movement of the user's arm will be described.

[0049] Figure 1 An example of an information processing device that operates based on the movement of a user's arm is shown. The information processing device includes a sensor 500 and a display 600. The sensor 500 is, for example, a camera. The sensor 500 continuously captures the user UR. The user UR has an arm 71. Here, the arm 71 includes not only an upper arm and a forearm, but also a hand. The hand also includes a fingertip 72. The information processing device controls the data displayed on the display 600 based on the position of the fingertip 72 of the user UR.

[0050] The information processing device sets a touch determination surface AR100 for accepting operations on a device (e.g., display 600) and a detection effective area AR200 for detecting the movement of the user UR's fingertip 72. In this example, the touch determination surface AR100 is set as a plane parallel to the display surface of the display 600.

[0051] When fingertip 72 enters a range closer to display 600 than touch determination surface AR100 , the information processing device detects that an operation instruction has been given to the information processing device, executes processing based on the operation instruction, and displays the execution result on display 600 .

[0052] Next, in Figure 2A and Figure 2B In the following, an example of user UR giving operation instructions for an information processing device is described. Figure 2A 1 is a diagram showing an operation example of the user UR as viewed from above. Figure 2B 1 is a diagram showing an operation example of the user UR as viewed from the side.

[0053] like Figure 2A and Figure 2B As shown, when the fingertip 72 enters the range of the touch determination surface AR100 (in other words, the fingertip 72 is located closer to the display 600 than the touch determination surface AR100), the information processing device determines that a specified location of the display 600 is touched. The touch determination surface AR100 is an area for accepting operations performed on the display 600. Hereinafter, the determination by the information processing device that the fingertip 72 of the user UR touches a specified location of the display 600 is sometimes referred to as touch detection. In addition, when the user UR moves the fingertip 72 along the trajectory TRK in a direction away from the touch determination surface AR100, the information processing device 100 determines that the state of touching the specified location of the display 600 has been transitioned to the released state.

[0054] Next, use Figure 3A and Figure 3B The relationship between the range in which the user UR can move the fingertip 72, the touch determination surface AR100, and the detection effective area AR200 is shown. In addition, in this example, it is assumed that the display 600 is mounted on the vehicle, and it is assumed that the user UR is sitting on the seat on the right. Therefore, the user UR is located on the right side of the figure. Here, in the detection effective area AR200, the finger of the user UR can be detected. Finger detection refers to detecting the position where the finger of the user UR is present. When the finger of the user UR is located in the detection effective area AR200, finger detection can be performed.

[0055] like Figure 3A As shown, the range in which the user UR can move the fingertip 72 is the range obtained by combining the area AR300 , the area AR400 , and the area AR500 .

[0056] The area AR300 is located in the range where the user UR can move the fingertip 72 and is closer to the display 600 than the touch determination surface AR100. Figure 3B As shown, the information processing device 100 can perform touch detection and finger detection in the area AR300.

[0057] The area AR400 is located in the range where the user UR can move the fingertip 72, and is farther from the display 600 than the touch determination surface AR100, but is included in the detection effective area AR200. Figure 3B As shown, the area AR400 is an area where touch detection cannot be performed but finger detection can be performed.

[0058] The area AR500 is located in the range where the user UR can move the fingertip 72, is farther from the display 600 than the touch determination surface AR100, and is not included in the detection effective area AR200. Figure 3B As shown, area AR500 is an area where neither touch detection nor finger detection can be performed.

[0059] exist Figure 3A In the example shown, there is a problem that the area where touch detection can be performed within the range where the user UR can move the fingertip 72 is small, that is, the range where the user UR can operate the display 600 is small.

[0060] As a method of solving this problem, it is conceivable to bring the touch determination plane AR100 closer to the position of the user UR. In the following, a range closer to the display 600 than the touch determination plane AR100 may be referred to as a touch detectable range.

[0061] Here, Figure 4A and Figure 4B2 shows an example in which the touch determination plane AR100 is brought close to the position of the user UR. Figure 4A As shown in FIG. 1 , the fingertip 72 of the user UR can easily reach the touch detection range. On the other hand, for example, the part of the user UR other than the fingertip 72, such as the knee of the user UR, may enter the touch detection range, and the information processing device may mistakenly detect the movement of the part other than the fingertip 72 of the user UR as a touch operation. Figure 4B As shown, the user UR is operating the smartphone SP or the like and has no intention of operating the display 600 , but the information processing device mistakenly detects the movement of the fingertip 72 of the user UR as a touch operation.

[0062] Although the touch determination surface AR100 is simply brought close to the user UR in this manner, there is a possibility of erroneous detection. Therefore, the information processing device according to the first embodiment sets a virtual operation area that is easy for the user UR to operate.

[0063] Figure 5 1 is a diagram showing a schematic configuration of an information system including a control device 1 as an information processing device according to the first embodiment. The information processing device according to the first embodiment includes a sensor 50 , a control device 1 , and a display 60 .

[0064] The sensor 50 is, for example, a camera device. As an example, the sensor 50 is a visible light camera. The sensor 50 outputs an image obtained by photographing the user UR to the control device 1. In addition, the sensor 50 is not limited to a visible light camera, and may be, for example, a CCD camera or a CMOS camera. The sensor 50 is an example of a camera unit. The sensor 50 continuously performs a camera process and outputs an image to the control device 1.

[0065] The display 60 is a display unit for displaying various data. The display 60 is an example of a device.

[0066] The control device 1 executes processing of data displayed on the display 60 according to the movement of the fingertip 72 of the user UR.

[0067] The control device 1 includes a control unit 10 and a storage unit 30. The control unit 10 is configured as, for example, a CPU (Central Processing Unit), and comprehensively controls the operations of each unit of the control device 1. The control unit 10 includes an image acquisition unit 11, an image processing unit 12, a landmark extraction unit 13, a detection effective area setting unit 14, a touch determination surface setting unit 15, a 3D coordinate calculation unit 16, a 2D coordinate conversion unit 17, a determination unit 18, a function execution unit 19, and a display output unit 20. The control device 1 includes, for example, a processor and a memory, and the functions of the control unit 10 and each functional block of the control unit 10 are realized by the processor executing a program stored in the memory. The CPU is an example of a processor. The storage unit 30 is an example of a memory.

[0068] The storage unit 30 stores various information. The storage unit 30 is implemented by hardware such as a memory or a storage device for storing information (in other words, data). Specifically, the storage unit 30 stores reference coordinate information 31, detection effective area information 32, and touch determination surface information 33. The reference coordinate information 31 is the 3D coordinates of the setting position of the sensor 50, the information of the installation angle of the sensor 50, the 3D coordinates of the position of the display 60, etc. The detection effective area information 32 is coordinate information indicating the detection effective area. The detection effective area is an area for detecting the position change of the fingertip 72 of the user UR. The touch determination surface information is coordinate information indicating the touch determination surface. The touch determination surface refers to the area used to accept operations performed on the display 60.

[0069] The image acquisition unit 11 acquires an image from the sensor 50. The image acquisition unit 11 is an example of an image acquisition unit. The image processing unit 12 processes the area in the image acquisition unit 11 where the user UR is not photographed as a removal object. The landmark extraction unit 13 detects the landmark part of the user UR from the image in which the user UR is photographed, and extracts the part that becomes the landmark. Here, the landmark refers to the eyes and three-dimensional coordinates of the shoulders of the user UR, and the arm part of the user. In addition, the landmark extraction unit 13 can also extract the length of the arm part of the user. In addition, the landmark extraction unit 13 can also extract information representing the field of view of the user UR based on the position of the eyes of the user UR.

[0070] The detection effective area setting unit 14 sets the detection effective area and registers the detection effective area information 32 indicating the coordinates of the detection effective area in the storage unit 30. The method of setting the detection effective area by the detection effective area setting unit 14 will be described later. The detection effective area setting unit 14 is an example of a setting unit. The detection effective area is an example of a detection area.

[0071] The touch determination surface setting unit 15 sets a touch determination surface and registers the touch determination surface information 33, which is the coordinates representing the coordinates of the touch determination surface, in the storage unit 30. The touch determination surface setting unit 15 is an example of a setting unit. The method for the touch determination surface setting unit 15 to set the touch determination surface will be described later. The touch determination surface is an example of a virtual operation area.

[0072] The 3D coordinate calculation unit 16 calculates the three-dimensional coordinates of the fingertip of the user UR. The 2D coordinate conversion unit 17 performs conversion from the three-dimensional coordinates of the fingertip of the user UR to two-dimensional coordinates on the display 60.

[0073] The determination unit 18 determines whether to touch and process the content based on the touch determination surface and the three-dimensional coordinates of the fingertip. The processing content is, for example, touch, flick, drag, etc.

[0074] The function execution unit 19 executes the processing content based on the result determined by the determination unit 18. The display output unit 20 causes the result obtained by executing the processing content to be displayed on the display 60. The determination unit 18, the function execution unit 19, and the display output unit 20 are examples of an operation unit.

[0075] (Touch determination surface setting method)

[0076] Next, the method for the touch determination surface setting unit 15 to set the touch determination surface will be described. First, Fig. 6A and Figure 6B are used to illustrate an example of setting the touch determination surface. As a premise, the image acquisition unit 11 acquires an image including the user UR. Moreover, the landmark extraction unit 13 extracts the shoulder part, the arm part, and the eye part of the user UR from the image including the user UR. In addition, the landmark extraction unit 13 may also extract the position of the elbow of the arm part of the user UR.

[0077] Fig. 6A is a diagram showing the touch determination surface observed from above. Figure 6B is a diagram showing the touch determination surface observed from the side.

[0078] As Fig. 6A and Figure 6B shown, the touch determination surface setting unit 15 sets the touch determination surface AR1 based on the following spherical formula (1), and the spherical surface has the center position S(xs, ys, zs) at the shoulder part and is formed by the distance L based on the length of the arm.

[0079] (X - xs)2 + (Y - ys)2 + (Z - zs))2 = L2 (1)

[0080] In addition, the above-mentioned distance L is a distance based on the length of the arm 71, for example, a length less than the entire length of the arm 71. The distance L is, for example, the entire length of the arm 71. In this way, the touch determination surface setting unit 15 sets the touch determination surface AR1 based on the position of the shoulder of the user UR and the length of the arm 71 of the user UR. In addition, the touch determination surface setting unit 15 sets the touch determination surface based on the spherical range by setting the touch determination surface AR1 based on the above-mentioned formula (1). The touch determination surface AR1 can be the entire spherical surface represented by formula (1), such as Fig. 6A and Figure 6B For example, based on the movable range of the fingertip 72 of the user UR, a part of the spherical surface represented by the equation (1) may be set as the touch determination surface AR1.

[0081] In addition, the touch determination surface setting unit 15 may further set the touch determination surface AR1 based on the visual field range of the user UR. Fig. 7A and Figure 7B A method of setting the touch determination surface AR1 by the touch determination surface setting unit 15 will be described in detail below.

[0082] Fig. 7A is a diagram showing a touch determination surface viewed from above, Figure 7B : is a diagram showing the touch determination surface viewed from the side. Here, the range from the position of the user's eyes to the end of the display 60 is set as the field of view. Specifically, when viewed from above, the range surrounded by plane L11, plane L12 and the display 60 is set as the field of view, wherein plane L11 is a plane containing the position of the user UR's eyes and the right edge of the display 60, and plane L12 is a plane containing the position of the user UR's eyes and the left edge of the display 60. Fig. 7A In FIG. 6 , plane L11 and plane L12 are shown as straight lines. In addition, when viewed from the side, the range surrounded by plane L13, plane L14, and display 60 is set as the field of view, where plane L13 is a plane including the position of the user UR's eyes and the upper edge of the display 60, and plane L14 is a plane including the position of the user UR's eyes and the lower edge of the display 60. Figure 7B In FIG. 1 , plane L13 and plane L14 are shown as straight lines. The touch determination surface setting unit 15 sets the touch determination surface AR1 based on the arm 71 and the visual field as described above. Specifically, the touch determination surface setting unit 15 sets only the portion of the spherical surface represented by equation (1) that is included in the visual field as the touch determination surface AR1. Fig. 7A As shown, the position P3 in the touch determination surface AR1 corresponds to the position P1 of the display 60, and the position P4 in the touch determination surface AR1 corresponds to the position P2 of the display 60. Figure 7B As shown, the position P7 on the touch determination surface AR1 corresponds to the position P5 on the display 60 , and the position P8 on the touch determination surface AR1 corresponds to the position P6 on the display 60 .

[0083] use Fig. 8A and Figure 8B Another example of setting the touch determination surface AR1 will be described. Fig. 8A is a diagram showing a touch determination surface viewed from above, Figure 8B FIG. 4 is a diagram showing a touch determination surface viewed from the side. Fig. 8A and Figure 8B As shown, the touch determination surface setting unit 15 may also set the touch determination surface AR1 with the elbow portion of the arm as the center. In this case, the touch determination surface setting unit 15 sets the touch determination surface AR1 based on the length from the elbow to the hand in the arm 71. The length from the elbow to the hand refers to the length from the elbow to the fingertips.

[0084] use Fig.9A and Fig. 9B Another example of setting the touch determination surface AR1 will be described. Fig.9A is a diagram showing a touch determination surface viewed from above, Fig. 9B FIG. 4 is a diagram showing a touch determination surface viewed from the side. Fig.9A and Fig. 9B As shown, the touch determination surface setting unit 15 may set the touch determination surface AR1 with the shoulder and elbow portions of the arm as the center.

[0085] In addition, after the touch determination surface setting unit 15 sets the touch determination surface AR1, when the determination unit 18 determines that the fingertip 72 of the user UR is in a state of passing through the touch determination surface AR1 (in other words, in a touch detection state) based on the positional relationship between the position of the finger part of the hand of the user UR acquired from the image and the touch determination surface AR1, the function execution unit 19 executes the function and the display output unit 20 displays the execution result on the display 60.

[0086] In the control device 1 according to the first embodiment, the image acquisition unit 11 acquires an image obtained by photographing the user, the landmark extraction unit 13 detects the arm portion of the user UR, and the touch determination surface setting unit 15 sets the touch determination surface AR1 based on the length of the arm with the shoulder portion as the center position. When the determination unit 18 determines that the touch detection state is in place based on the positional relationship between the position of the fingertip 72 of the user UR acquired from the image and the touch determination surface AR1, the function execution unit 19 executes the function, and the display output unit 20 displays the execution result on the display 60.

[0087] As described above, the control device 1 sets the touch determination surface AR1 based on the length of the arm of the user UR, and thus can set the touch determination surface AR1 according to the user's motion. In other words, it can set a virtual operation area that is easy for the user to operate.

[0088] In addition, the control device 1 sets the touch determination surface AR1 based on the length of the arm 71 with the shoulder as the center position. In other words, the touch determination surface AR1 is set based on the movable range of the arm 71. By setting the touch determination surface AR1 based on the movable range of the arm 71 in this way, compared with Figure 1 In the case where the touch determination surface AR600 is set parallel to the display 600 as shown in FIG. 8 , unintentional touch / release operations are less likely to be performed.

[0089] In addition, the touch determination surface setting unit 15 can set the touch determination surface AR1 based on the length from the elbow to the hand of the user UR, or can set the touch determination surface AR1 based on the overall length of the arm 71 of the user UR. In this case, the touch determination surface AR1 corresponding to the user's action can also be set. In addition, the touch determination surface setting unit 15 can also set the touch determination surface AR1 further based on the field of view of the user UR. In this case, the control device 1 is further limited to the field of view of the user UR, so that the touch determination surface AR1 that is easier for the user to operate can be set.

[0090] (Second Embodiment)

[0091] Next, a control device 1 according to a second embodiment will be described. In the control device 1 according to the second embodiment, a detection effective area that is easy for a user to operate is set.

[0092] (Detection effective area setting method)

[0093] The following describes a method for setting the effective detection area by the effective detection area setting unit 14. Fig. 10A and Fig. 10B The following is an example of setting the detection effective area. As a premise, the image acquisition unit 11 acquires an image including the user UR. Furthermore, the landmark extraction unit 13 extracts the shoulder, arm, and eye of the user UR from the image including the user UR. In addition, the landmark extraction unit 13 may also extract the position of the elbow of the arm of the user UR.

[0094] Fig. 10A 2 is a diagram showing the positional relationship between the user UR and the display 60 as viewed from above. Fig. 10B 1 is a diagram showing the positional relationship between the user UR and the display 60 when viewed from the side.

[0095] like Fig. 10A and Fig. 10B As shown in Fig. 10B , the detection effective area setting unit 14 sets the detection effective area AR2 based on the following equations (2) and (3) of the sphere, where the sphere has the shoulder part as the center position S(xs, ys, zs) and is composed of the distance L1 based on the length of the arm and the distance L2 based on the length of the arm.

[0096] (X-xs)2 + (Y-ys) 2 + (Z-zs)2 ≥ L12 (2)

[0097] (X - xs)2 + (Y-ys)2 + (Z-zs))2 ≤ L22 (3)

[0098] The detection effective area setting unit 14 sets the area that satisfies the above equations (2) and (3) as the detection effective area AR2. In addition, L1 < L2. Also, when showing the relationship between the distance L, the distance L1, and the distance L2 described in the first embodiment, there is L1 ≤ L ≤ L2. The distance L1 is, for example, a length less than or equal to the overall length of the arm 71. The distance L1 is, for example, the length from the shoulder to the elbow in the arm. The distance L2 is, for example, a length less than or equal to the overall length of the arm 71. The distance L2 is, for example, the overall length of the arm 71.

[0099] In this way, the detection effective area setting unit 14 sets the detection effective area AR2 based on the length of the arm 71 of the user UR. The detection effective area AR2 can be the entire area represented by equations (2) and (3), as Fig. 10A and Fig. 10B shown, or it can be a part of the area represented by equations (2) and (3). For example, based on the movable range of the fingertip 72 of the user UR, a part of the area represented by equations (2) and (3) can be set as the detection effective area AR2.

[0100] In addition, similar to the example of the first embodiment, the detection effective area setting unit 14 can also set the detection effective area AR2 with the elbow part as the center position, and can also set the detection effective area AR2 with both the shoulder and the elbow as fulcrums.

[0101] In addition, similar to the example of the first embodiment, the detection effective area setting unit 14 can also further set the detection effective area AR2 based on the visual field range of the user UR.

[0102] Next, use Fig.11 to illustrate the processing procedure of the control device 1 according to the second embodiment. Fig.11 is a flowchart showing the processing procedure of the control device 1 according to the second embodiment.

[0103] First, the landmark extraction unit 13 determines whether a user is detected from the image acquired by the image acquisition unit 11 (step S1). If the landmark extraction unit 13 does not detect a user from the image (step S1: "No"), the process returns to step S1. If the landmark extraction unit 13 extracts a user from the image (step S1: "Yes"), the landmark extraction unit 13 detects landmarks such as eyes, shoulders, and elbows, and calculates their 3D coordinates (step S2).

[0104] Furthermore, the landmark extraction unit 13 calculates the length of the user's arm based on the position of the landmark (step S3 ).

[0105] The detection effective area setting unit 14 sets the detection effective area based on the landmark position and the length of the user's arm (step S4). In addition, the touch determination surface setting unit 15 sets the touch determination surface based on the landmark position and the length of the user's arm (step S5). The 3D coordinate calculation unit 16 determines whether the fingertip 72 of the user UR is detected on the image (step S6). In step S6, when the 3D coordinate calculation unit 16 does not detect the fingertip of the user UR from the image (step S6: "No"), the processing enters step S1. When the 3D coordinate calculation unit 16 detects the fingertip of the user UR from the image (step S6: "Yes"), it calculates the 3D coordinates of the fingertip (step S7). The determination unit 18 determines whether the 3D coordinates of the fingertip are within the detection effective area AR2 (step S8). When the determination unit 18 determines that the fingertip is not within the detection effective area AR2 (step S8: "No"), the processing enters step S7.

[0106] In addition, when the determination unit 18 determines in step S8 that the 3D coordinates of the fingertip are within the detection effective area AR2 (step S8: "Yes"), it determines whether the fingertip has passed through the touch determination surface (step S9). When the determination unit 18 determines in step S9 that the fingertip has passed through the touch determination surface AR1 (step S9: "Yes"), it determines the operation corresponding to the movement of the fingertip. The operation corresponding to the movement of the fingertip refers to, for example, a touch operation. As specific examples of touch operations, click operations, drag operations, and slide operations can be cited. The function execution unit 19 executes the determined operation (step S10). When the determination unit 18 determines in step S9 that the fingertip has not passed through the touch determination surface AR1 (step S9: "No"), it determines the operation corresponding to the movement of the fingertip. The operation corresponding to the movement of the fingertip refers to, for example, a release operation or a pointer movement operation. The function execution unit 19 executes the determined operation (step S11).

[0107] In the control device 1 according to the second embodiment, the detection effective area setting unit 14 sets the detection effective area AR2 based on the length of the arm with the shoulder as the center position. The determination unit 18 determines the operation corresponding to the movement of the fingertip based on the positional relationship between the position of the fingertip 72 of the user UR acquired from the image and the detection effective area AR2, the function execution unit 19 executes the function corresponding to the operation, and the display output unit 20 displays the execution result on the display 60.

[0108] As described above, the control device 1 sets the effective detection area AR2 based on the length of the arm of the user UR, and thus can set the effective detection area AR2 according to the user's motion. In other words, it can set the effective detection area that is easy for the user to operate.

[0109] In addition, if Fig.12 As shown, the control device 1 sets the detection effective area AR2 based on the length of the arm 71 of the user UR1. Therefore, even if the user UR2 moves around the display 60, the user UR2 is not included in the detection effective area AR2. This can prevent the control device 1 from erroneously detecting the movement of another user.

[0110] (Third Embodiment)

[0111] Next, a control device 1 according to a third embodiment will be described. The control device 1 according to the third embodiment sets a detection effective area and a touch determination surface for each of a plurality of users, and performs touch determination based on the priority of the users.

[0112] The image acquisition unit 11 acquires an image in which a plurality of users are photographed. The landmark extraction unit 13 extracts the landmarks of each of the plurality of users. The detection effective area setting unit 14 sets the detection effective area AR2 of each user based on the length of each user's arm. In addition, the touch determination surface setting unit 15 sets the touch determination surface AR1 of each user based on the length of each user's arm.

[0113] When a plurality of users are captured, the control device 1 sets a user with a high priority. The determination unit 18 operates the display 60 based on the positional relationship between the fingertip 72 of the user with the high priority and the touch determination surface AR1.

[0114] Here, in Fig.13 2 shows an example of the positional relationship between a plurality of users and the display 60. Fig.13 As shown, when the user UR1 and the user UR2 are near the display 60, the control device 1 sets the touch determination surface AR1a and the detection effective area AR2a of the user UR1. In addition, the control device 1 sets the touch determination surface AR1b and the detection effective area AR2b of the user UR2.

[0115] Furthermore, the control device 1 increases the priority of the touch determination surface AR1a and the detection effective area AR2a of the user UR1 through a predetermined operation. As a result, the control device 1 sets the touch determination surface AR1a and the detection effective area AR2a to an effective state. Along with this, the control device 1 sets the touch determination surface AR1b and the detection effective area AR2b to an ineffective state.

[0116] Then, the control device 1 changes the display state of the display 60 based on the positional relationship between the position of the fingertip 72 of the user UR1 , the touch determination surface AR1 a , and the detection effective area AR2 a .

[0117] Next, use Fig.14 The processing procedure of the control device 1 according to the third embodiment will be described. Fig.14 It is a flowchart showing the processing procedure of the control device 1 according to the second embodiment.

[0118] First, the landmark extraction unit 13 determines whether a user is detected from the image acquired by the image acquisition unit 11 (step S21). If the landmark extraction unit 13 does not detect a user from the image (step S21: No), the process returns to step S21. If the landmark extraction unit 13 extracts a user from the image (step S21: Yes), the landmark extraction unit 13 detects landmarks such as eyes, shoulders, and elbows, and calculates their 3D coordinates (step S22).

[0119] Furthermore, the landmark extraction unit 13 calculates the length of the user's arm based on the position of the landmark (step S23 ).

[0120] The detection effective area setting unit 14 sets the detection effective area AR2 based on the landmark position and the length of the user's arm (step S24). In addition, the touch determination surface setting unit 15 sets the touch determination surface AR1 based on the landmark position and the length of the user's arm (step S25). In the case where there are users detected from the image for whom the detection effective area AR2 and the touch determination surface AR1 are not set (step S26: "Yes"), the processing enters step S22. In addition, in the case where the touch determination surface AR1 and the detection effective area AR2 of all users are set, that is, there are no unset users (step S26: "No"), the control device 1 sets the priority of multiple touch determination surfaces AR1 and detection effective areas AR2 (step S27).

[0121] The control device 1 sets the priority of the plurality of touch determination surfaces AR1 and the detection effective area AR2, for example, based on the content displayed on the display 60. The control device 1 may also set the priority after step S29 described later. For example, the control device 1 may set the touch determination surface AR1 and the detection effective area AR2 corresponding to the user who is located near the fingertip that has moved first among the fingertips whose 3D coordinates are calculated in step S29 to a high priority.

[0122] The 3D coordinate calculation unit 16 determines whether a fingertip 72 of a certain user is detected on the image (step S28). In step S28, when the 3D coordinate calculation unit 16 does not detect the fingertip of the user UR from the image (step S28: "No"), the processing enters step S21. When the 3D coordinate calculation unit 16 detects the fingertip of a certain user UR from the image (step S28: "Yes"), it calculates the 3D coordinates of the fingertip (step S29). The determination unit 18 determines whether the 3D coordinates of the fingertip are within the detection effective area AR2 with the highest priority (step S30). When the determination unit 18 determines that the fingertip is not within the detection effective area AR2 (step S30: "No"), the processing enters step S29.

[0123] In addition, when the determination unit 18 determines in step S30 that the 3D coordinates of the fingertip are within the detection effective area AR2 with the highest priority (step S30: "Yes"), it determines whether the fingertip has passed through the touch determination surface AR1 with the highest priority (step S31). When the determination unit 18 determines in step S31 that the fingertip has passed through the touch determination surface AR1 with the highest priority (step S31: "Yes"), it determines the operation corresponding to the movement of the fingertip. The operation corresponding to the movement of the fingertip refers to, for example, a touch operation. As specific examples of touch operations, click operations, drag operations, and slide operations can be cited. The function execution unit 19 executes the determined touch operation (step S32). When the determination unit 18 determines in step S31 that the fingertip has not passed through the touch determination surface AR1 (step S31: "No"), it determines the operation corresponding to the movement of the fingertip. The operation corresponding to the movement of the fingertip refers to, for example, a release operation or a pointer movement operation. The function execution unit 19 executes the determined operation (step S33).

[0124] As described above, the control device 1 according to the third embodiment can operate the display 60 based on the detection effective area AR2 and the touch determination surface AR1 with a higher priority when a plurality of detection effective areas AR2 and touch determination surfaces AR1 are set. Thus, the display 60 can be operated based on the action of the user that is more appropriate as the operation instruction.

[0125] (Fourth Embodiment)

[0126] Next, the control device 1 according to the fourth embodiment is described. The control device 1 according to the fourth embodiment removes an unnecessary area in which the user is not photographed from the image, and operates the display 60 based on the positional relationship between the position of the finger detected from the image after the unnecessary area is removed and the detection effective area AR2 and the touch determination surface AR1.

[0127] exist Fig.15 An example of the positional relationship between the user and the display 60 is shown. Fig.15 In the example shown, the range that the sensor 50 can capture includes area AR11 and area AR12. Area AR11 includes the detection effective area AR2 and the touch determination surface AR1. On the other hand, area AR12 does not include the detection effective area AR2 and the touch determination surface AR1. Therefore, even if the finger is located in area AR12, the control device 1 does not need to perform coordinate calculations on the finger. Therefore, the image processing unit 12 of the control device 1 performs a removal process to remove the portion of the image corresponding to area AR12. The image processing unit 12 performs the removal process, for example, by trimming off the portion corresponding to area AR12. By performing the removal process, the area in the image where coordinate calculations are performed, i.e., the coordinate calculation area, can be reduced.

[0128] In addition, the image processing unit 12 may perform the removal process by removing the portion of the image other than the portion in which the user is photographed. Fig.16A and Fig. 16B An example of removal processing will be described.

[0129] The image processing unit 12 only retains Fig.16A The removal process is to remove the area AR111 of the image where the user is photographed and the area AR112 of the image where the user is not photographed. The result of the removal process is as follows: Fig. 16B As shown in FIG. 1 , an image can be obtained in which only the area AR111 of the portion where the user is photographed remains. As a result, the coordinate calculation area is only the area AR111.

[0130] Next, use Fig.17 The processing procedure of the control device 1 according to the fourth embodiment will be described. Fig.17 It is a flowchart showing the processing procedure of the control device 1 according to the fourth embodiment.

[0131] First, the landmark extraction unit 13 determines whether a user is detected from the image acquired by the image acquisition unit 11 (step S41). If the landmark extraction unit 13 does not detect a user from the image (step S41: No), the process returns to step S1. If the landmark extraction unit 13 extracts a user from the image (step S41: Yes), the landmark extraction unit 13 detects landmarks such as eyes, shoulders, and elbows, and calculates their 3D coordinates (step S42).

[0132] Furthermore, the landmark extraction unit 13 calculates the length of the user's arm based on the position of the landmark (step S43 ).

[0133] The detection effective region setting unit 14 sets the detection effective region based on the landmark position and the length of the user's arm (step S44 ). In addition, the touch determination surface setting unit 15 sets the touch determination surface based on the landmark position and the length of the user's arm (step S45 ).

[0134] The image processing unit 12 reduces the coordinate calculation area of ​​the image by removal processing (step S46). The 3D coordinate calculation unit 16 determines whether the fingertip 72 of the user UR is detected in the coordinate calculation area on the image (step S47). In step S47, when the 3D coordinate calculation unit 16 does not detect the fingertip of the user UR in the coordinate calculation area on the image (step S47: "No"), the processing enters step S41. When the 3D coordinate calculation unit 16 detects the fingertip of the user UR in the coordinate calculation area on the image (step S47: "Yes"), the 3D coordinates of the fingertip are calculated (step S48). The determination unit 18 determines whether the 3D coordinates of the fingertip are within the detection effective area AR2 (step S49). When the determination unit 18 determines that the fingertip is not within the detection effective area AR2 (step S49: "No"), the processing enters step S48.

[0135] In addition, when the determination unit 18 determines in step S49 that the 3D coordinates of the fingertip are within the detection effective area AR2 (step S49: "Yes"), it determines whether the fingertip has passed through the touch determination surface (step S50). When the determination unit 18 determines in step S50 that the fingertip has passed through the touch determination surface AR1 (step S50: "Yes"), it determines the operation corresponding to the movement of the fingertip. The operation corresponding to the movement of the fingertip refers to, for example, a touch operation. As specific examples of touch operations, click operations, drag operations, and slide operations can be cited. The function execution unit 19 executes the determined operation (step S51). When the determination unit 18 determines in step S50 that the fingertip has not passed through the touch determination surface AR1 (step S50: "No"), it determines the operation corresponding to the movement of the fingertip. The operation corresponding to the movement of the fingertip refers to, for example, a release operation or a pointer movement operation. The function execution unit 19 executes the determined operation (step S52).

[0136] As described above, the control device 1 according to the fourth embodiment removes positions in the image that are greatly deviated from the positions of the detection effective area AR2 and the touch determination surface AR1 from the coordinate calculation targets. This can reduce the processing load on the control device 1 .

[0137] (Variation Example)

[0138] In the above-mentioned embodiment, although not specifically described, the position of the sensor 50 may be appropriately configured based on the positional relationship between the user UR and the display 60. For example, Fig.18 As shown, the sensor 50 may be arranged in front of the display 60. In this case, the detection effective area AR2 and the touch determination surface AR1 can be more reliably included in the sensing area AR3 which is the range captured by the sensor 50.

[0139] In addition, if Fig.19 As shown, the sensor 50 may be arranged toward the direction of the user UR. For example, when the user UR is located on the right side relative to the display 60, the sensor 50 may be arranged to face the right. In this case, the control device 1 can concentrate the sensing area AR3 around the user UR, thereby reducing the possibility that the control device 1 erroneously detects the motion of other users.

[0140] In addition, if Fig. 20 As shown, the sensor 50 may be arranged in front of the display 60 and may be arranged to face the direction of the user UR.

[0141] In addition, the shape of the display 60 is arbitrary. For example, Fig.21 As shown, a curved display 60 may also be used.

[0142] In addition, the control device 1 may also set the detection effective area AR2 and the touch determination surface AR1 corresponding to a plurality of devices. Fig. 22 As shown, the control device 1 can set the detection effective area AR2 and the touch determination surface AR1 corresponding to all the devices in the device 61a, the device 61b, and the device 61c. In this case, the control device 1 can determine the field of view for each device, and perform operation control based on the positional relationship between the position of the user UR's finger and the detection effective area AR2 and the touch determination surface AR1, and the field of view of each device.

[0143] In addition, the control device 1 may also set the detection effective area AR2 and the touch determination surface AR1 corresponding to each device 61 in consideration of the field of view of each device 61. Fig.23As shown, the control device 1 sets the effective detection areas AR2a to AR2c and the touch determination surfaces AR1a to AR1c in consideration of the field of view of each device 61. In this way, the control device 1 sets the touch determination surfaces so that the touch determination surfaces AR1a to AR1c do not overlap.

[0144] In the above description, the control device 1 sets the detection effective area AR2 and the touch determination surface AR1 based on the position and length of the right arm 71, but the detection effective area AR2 and the touch determination surface AR1 may also be set based on the position and length of the left arm 71. Fig.24A and Fig. 24B As shown, the detection effective area AR2 and the touch determination surface AR1 ( FIG. 1 ) for the left arm 71 may be set based on the position and length of the left arm 71. Fig.24A ), and the detection effective area AR2 and the touch determination surface AR1 for the right arm 71 based on the position and length of the right arm 71 ( Fig. 24B ).

[0145] In this case, after determining whether the finger detected from the image is the finger of the right arm or the finger of the left arm, the determination unit 18 determines the positional relationship between the detected finger and the corresponding detection effective area AR2 and touch determination surface AR1.

[0146] In addition, you can also Fig.25A As shown in FIG. 1 , the area formed by combining the effective detection area AR2a corresponding to the left arm 71 and the effective detection area AR2b corresponding to the right arm 71 is set as the effective detection area. Fig.25B As shown, you can also set Fig.25A The detection effective area AR2a and the detection effective area AR2b are shown as the detection effective area AR2c.

[0147] In this case, regardless of whether the finger detected from the image is a finger of the right arm or a finger of the left arm, the determination unit 18 determines the positional relationship between the position of the finger and the detection effective area AR2c. In addition, regarding the determination of the positional relationship between the finger and the touch determination surface AR1, the determination unit 18 distinguishes whether the finger is a finger of the left arm or a finger of the right arm, and determines the positional relationship between the corresponding touch determination surface AR1 and the finger.

[0148] Regarding the above-mentioned embodiment, the following contents are disclosed.

[0149] An information processing device comprising:

[0150] An image acquisition unit that acquires an image obtained by photographing the user;

[0151] a detection unit that detects an arm portion of a user based on the image;

[0152] a setting unit that sets a detection area that is an area for detecting a position change of the user's fingertip based on the position of the user's arm and the length of the user's arm detected by the detection unit;

[0153] An operating unit operates the device based on the positional relationship between the user's fingertip and the detection area according to the image.

[0154] The embodiments of the present disclosure are described above, but the above embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the subject matter of the invention. These new embodiments and their modifications are included in the scope and subject matter of the invention, and are included in the invention described in the claims and the scope of its equivalent. In addition, the constituent elements involved in different embodiments and modifications may be appropriately combined.

[0155] In addition, the expression "···part" in the above-mentioned embodiments may be replaced by other expressions such as "···circuitry", "···component", "···device", "···unit" or "···module".

[0156] In the above-mentioned embodiments, the present disclosure has been described by taking an example in which the present disclosure is configured using hardware. However, the present disclosure can also be realized by software in cooperation with hardware.

[0157] In addition, each functional block used in the description of each embodiment is typically implemented as an LSI as an integrated circuit. The integrated circuit controls each functional block used in the description of the above embodiment, and the integrated circuit may have an input terminal and an output terminal. They can be single-chipped individually or in a manner that includes part or all of them. It is set as LSI here, but depending on the degree of integration, it is sometimes called IC, system LSI, super LSI, and ultra LSI.

[0158] In addition, the method of integrated circuitization is not limited to LSI, and can also be implemented using dedicated circuits or general-purpose processors and memories. It is also possible to use FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, and reconfigurable processors that can reconfigure the connections or settings of circuit cells inside LSI.

[0159] Furthermore, if integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology or other derived technologies, it is of course possible to use this technology to integrate functional blocks. Application of biotechnology is a possible possibility.

[0160] In addition, the effects in the embodiments described in this specification are merely illustrative and non-limiting, and other effects may also be present.

[0161] Description of Reference Numerals

[0162] 1: control device; 10: control unit; 11: image acquisition unit; 12: image processing unit; 13: landmark extraction unit; 14: detection effective area setting unit; 15: touch judgment surface setting unit; 16: 3D coordinate calculation unit; 17: 2D coordinate conversion unit; 18: judgment unit; 19: function execution unit; 20: display output unit; 50: sensor; 60: display.

Claims

1. An information processing device comprising: An image acquisition unit that acquires an image obtained by photographing the user; a detection unit that detects an arm portion of a user based on the image; a setting unit that sets a virtual operation area for accepting an operation on a device based on the position of the user's arm and the length of the user's arm detected by the detection unit; and An operating unit operates a device based on the positional relationship between the user's fingertip and the virtual operating area according to the image.

2. The information processing device according to claim 1, wherein: The setting unit sets the virtual operation area based on the entire length of the user's arm detected by the detection unit.

3. The information processing device according to claim 1, wherein: The setting unit sets the virtual operation area based on the length from the elbow to the hand of the arm of the user detected by the detection unit.

4. The information processing device according to claim 1, wherein: The detection unit detects a user's field of view based on the image; The setting unit further sets the virtual operation area based on a field of view of the user.

5. The information processing device according to claim 1, wherein: The setting unit sets a detection area as an area for detecting a position change of the user's fingertip based on the position of the user's arm and the length of the user's arm detected by the detection unit. The operating unit operates a device based on a positional relationship between a fingertip of the user and the detection area according to the image.

6. The information processing device according to claim 5, wherein: The detection unit detects a user's field of view based on the image; The setting unit further sets the detection area based on a visual field of the user.

7. The information processing device according to claim 1, wherein: The detection unit detects arm parts of a plurality of users detected by the detection unit, The setting unit sets a virtual operation area for each user based on the lengths of the arm parts of the plurality of users detected by the detection unit, The operating unit operates the device based on the positional relationship between a fingertip of one of the plurality of users and the virtual operating area having a high priority, according to the image.

8. The information processing device according to claim 1, wherein: The operation unit detects a fingertip of the user from an image obtained by removing an area of ​​the image in which the user is not captured, and operates a device based on a positional relationship between the detected hand and the virtual operation area.

9. The information processing device according to claim 1, wherein: A plurality of the devices are arranged in an array; The setting unit sets each virtual operation area so that the virtual operation areas of each device among the devices do not overlap.

10. The information processing device according to claim 1, wherein: The detection unit detects left and right arm parts of the user; The setting unit sets left and right virtual operation areas based on the lengths of left and right arms of the user.

11. The information processing device according to claim 1, wherein: The setting unit sets the virtual operation area based on a spherical range obtained based on the length of the user's arm and the position of the user's shoulder detected by the detection unit.

12. An information processing system comprising a camera unit, an information processing device and a display unit, wherein: The camera unit captures an image containing the user, The information processing system comprises: an image acquisition unit, which acquires the image; a detection unit that detects an arm portion of a user based on the image; a setting unit that sets a virtual operation area for accepting an operation performed on the display unit based on the position of the user's arm portion and the length of the user's arm portion detected by the detection unit; as well as An operating unit operates a display unit based on a positional relationship between a fingertip of the user and the virtual operating area according to the image.

13. The information processing system according to claim 12, wherein: The imaging unit is arranged based on a positional relationship between the position of the user and the display unit.

14. An information processing method, executed by an information processing device, the information processing method comprising: An image acquisition step, acquiring an image obtained by photographing the user; a detection step of detecting an arm portion of the user based on the image; a setting step of setting a virtual operation area for accepting operations on a device based on the position of the user's arm and the length of the user's arm detected in the detecting step; as well as An operation step is to operate the device based on the positional relationship between the user's fingertip and the virtual operation area according to the image.

Citation Information

Cited By

  • Multi-user cooperative interaction method and system for best machine

    CN122308699A

  • A method and system for multiple user interaction of a BFF machine

    CN122308699B