Operating hand tracking method and device, equipment and storage medium

By obtaining the head and shoulder positions of the operator and operator in the first image, determining the target wrist position, and updating the operator information, the problem of low tracking accuracy of operators in the prior art is solved, and the multi-dimensional Accurate tracking and smooth gesture control in human scenes.

CN118982558APending Publication Date: 2024-11-19SHENZHEN HONGHE INNOVATION INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411049922.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The prior art has low accuracy in operator tracking, especially in rapid inflection point motion or multi-user hand scenarios, which are prone to loss or error tracking.

Method used

By acquiring the operator information and operator information in the first image, including the head and shoulder positions, the target wrist position is determined, and the operator information is updated according to the wrist position, the tracking accuracy is improved.

Benefits of technology

It realizes accurate tracking of the operator, reduces the error tracking rate, and can accurately lock the operator in multiple scenarios, improving the smoothness of gesture control and use sensitivity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982558A_ABST
    Figure CN118982558A_ABST
Patent Text Reader

Abstract

The invention discloses a manipulator tracking method and device, equipment and a storage medium, and belongs to the field of human-computer interaction. The method comprises the following steps: acquiring a first image; if the first gesture exists in the first image, operator information and operator information are obtained according to an operator corresponding to the first gesture in the first image, the operator information comprises the position of the operator, and the operator information comprises the head position and the shoulder position of the operator to which the operator belongs. Then, acquiring a second image, determining the position of each user hand in one or more user hands in the second image, determining a target wrist position in the second image according to the operator information, and determining an operator in the second image according to the position of each user hand in one or more user hands in the second image and the target wrist position; and the operator information and the operator information are updated accordingly. According to the invention, the manipulator can be accurately tracked.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to an operator tracking method, device, equipment and storage medium. Background Art

[0002] With the continuous development of science and technology, gesture control technology has been gradually applied to various fields. At present, electronic devices can analyze the captured images, identify the hand making the access gesture as the operator, and then track the operator to perform the corresponding control operation when the operator makes the control gesture.

[0003] In the related art, when the electronic device tracks the operator, it predicts the position of the operator in the next frame of image based on the moving speed and position of the operator in the previous frame of image. However, the accuracy of this method is low. Summary of the invention

[0004] The present application provides a method, device, equipment and storage medium for tracking an operator, which are applied to electronic devices and can improve the accuracy of tracking an operator. The technical solution is as follows:

[0005] In a first aspect, a method for tracking an operator is provided, the method comprising:

[0006] acquiring a first image;

[0007] If the first gesture exists in the first image, acquiring operator information and operator information according to the operator corresponding to the first gesture in the first image, the operator information including the position of the operator, and the operator information including the head position and shoulder position of the operator to which the operator belongs;

[0008] Acquire a second image, where the second image is a frame of image acquired after the first image;

[0009] determining a position of each of the one or more user hands in the second image;

[0010] determining a target wrist position in the second image according to the operator information;

[0011] According to the position of each of the one or more user hands in the second image and the target wrist position, the operator in the second image is determined, and the operator information and the operator person information are updated.

[0012] In the present application, a first image is acquired; if there is a first gesture in the first image, the operator information and the operator information are acquired according to the operator corresponding to the first gesture in the first image, the operator information includes the position of the operator, and the operator information includes the head position and shoulder position of the operator to which the operator belongs. After that, a second image is acquired to determine the position of each user hand in one or more user hands in the second image. Then, the target wrist position in the second image is determined according to the operator information. Since the operator's head and shoulders usually do not undergo a large displacement in the gesture control process, the target wrist position can be accurately determined according to the operator information. And since the distance between the operator's wrist and hand is very close, the operator in the second image can be more accurately determined according to the position of each user hand in one or more user hands in the second image and the target wrist position, and the operator information and the operator information are updated accordingly. In this way, accurate tracking of the operator can be achieved.

[0013] Optionally, before acquiring the operator information and the operator information according to the operator corresponding to the first gesture in the first image, the method further includes:

[0014] If the target flag is logically false, determining whether the first gesture exists in the first image;

[0015] The method further comprises:

[0016] If the first gesture exists in the first image, the target flag is set to logical true.

[0017] Optionally, determining the target wrist position in the second image according to the operator information includes:

[0018] If the target flag is logically true, the target wrist position is determined according to the operator information.

[0019] Optionally, after acquiring the operator information and the operator information according to the operator corresponding to the first gesture in the first image, the method further includes:

[0020] Start the first count;

[0021] The step of determining the target wrist position in the second image according to the operator information includes:

[0022] If the timing duration of the first timing is greater than or equal to the first preset duration, the target wrist position is determined according to the operator information, and the first timing is reset.

[0023] Optionally, after determining the position of each user's hand among the one or more user's hands in the second image, the method further includes:

[0024] If the timing duration of the first timing is less than the first preset duration, the operator in the second image is determined according to the operator information and the position of each user hand of one or more user hands in the second image, and the operator information is updated.

[0025] Optionally, the operator information further includes a hand type of the operator, and before determining the target wrist position in the second image according to the operator information, the method further includes:

[0026] Acquire a target set, where the target set includes all user hands in the second image;

[0027] Deleting the user hands in the target set that are different in type from the hand in the operator hand information;

[0028] Deleting a user hand from the target set whose distance from the position in the operator hand information is greater than or equal to a preset distance;

[0029] The determining the operating hand in the second image according to the position of each of the one or more user hands in the second image and the target wrist position includes:

[0030] According to the position of each user's hand among all the user's hands in the target set and the target wrist position, the operating hand in the second image is determined from the user's hands in the target set.

[0031] Optionally, determining the target wrist position in the second image according to the operator information includes:

[0032] cropping the second image according to the head position and shoulder position of the operator to obtain a third image;

[0033] Perform wrist detection on the third image to obtain the target wrist position.

[0034] Optionally, the step of cropping the second image according to the head position and shoulder position of the operator to obtain the third image includes:

[0035] Determine a minimum rectangular area in the second image that includes the head position and shoulder position of the operator;

[0036] Expanding the minimum rectangular area according to a first ratio to obtain a first area;

[0037] The third image located within the first area is cropped from the second image.

[0038] Optionally, determining the operating hand in the second image according to the position of each user's hand among the one or more user's hands in the second image and the target wrist position includes:

[0039] The user hand of the one or more user hands in the second image whose distance to the target wrist position is the smallest is determined as the operating hand in the second image.

[0040] Optionally, after obtaining the operator information, the step further includes:

[0041] Start the second timer;

[0042] After determining the operator in the second image, the method further includes:

[0043] determining whether a second gesture is recognized according to the operator in the second image;

[0044] If the second gesture is not recognized and the timing duration of the second timing is greater than or equal to the second preset duration, the gesture control process is terminated.

[0045] Optionally, after determining whether the second gesture is recognized according to the operator in the second image, the method further includes:

[0046] If the second gesture is recognized, the control operation corresponding to the second gesture is executed.

[0047] In a second aspect, a device for tracking an operator's hand is provided, the device comprising:

[0048] A first acquisition module, used for acquiring a first image;

[0049] a second acquisition module, configured to acquire, if a first gesture exists in the first image, operator information and operator person information according to an operator corresponding to the first gesture in the first image, wherein the operator information includes a position of the operator, and the operator person information includes a head position and a shoulder position of an operator to which the operator belongs;

[0050] A third acquisition module is used to acquire a second image, where the second image is a frame of image acquired after the first image;

[0051] a first determination module, configured to determine a position of each of the one or more user hands in the second image;

[0052] A second determination module, configured to determine a target wrist position in the second image according to the operator information;

[0053] The third determination module is used to determine the operator in the second image according to the position of each user's hand of one or more users in the second image and the target wrist position, and update the operator information and the operator information.

[0054] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the operating hand tracking method described in the first aspect.

[0055] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the operating hand tracking method described in the first aspect is implemented.

[0056] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the steps of the operating hand tracking method described in the first aspect.

[0057] It can be understood that the beneficial effects of the second, third, fourth and fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 is a flow chart of an operator tracking method provided in an embodiment of the present application;

[0060] Figure 2 is a flow chart of another operator tracking method provided in an embodiment of the present application;

[0061] Figure 3 is a structural schematic diagram of an operator tracking device provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0064] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0065] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" that appear in the differences in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. In addition, the terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0066] The application scenarios of the embodiments of the present application are described below.

[0067] Gesture control technology has been gradually applied to gaming, smart home, healthcare and other fields due to its features of remote control, diverse application scenarios, timely feedback and dynamic response. Currently, electronic devices can analyze the captured images, identify the hand making the access gesture as the operator, and then track the operator to perform the corresponding control operation when the operator makes the control gesture.

[0068] In the related art, when the electronic device tracks the operator, it predicts the position of the operator in the next frame of the image based on the moving speed and position of the operator in the previous frame of the image. However, in some special cases, for example, when the operator moves at a rapid turning point, the operator may be lost through this method; for example, when there are many user hands near the operator, other user hands may be tracked incorrectly through this method. Therefore, the tracking accuracy of this method is low.

[0069] To this end, an embodiment of the present application provides an operator tracking method that can track the operator more accurately.

[0070] The operator tracking method provided in the embodiment of the present application is explained in detail below.

[0071] Figure 1 is a flow chart of a method for tracking an operator provided in an embodiment of the present application. Figure 1 , the method is applied to an electronic device, the method comprising the following steps:

[0072] Step 101: An electronic device acquires a first image.

[0073] The first image is a frame of image acquired by the electronic device. For example, the first image can be a frame of image captured by the electronic device through a camera.

[0074] Step 102: If there is a first gesture in the first image, the electronic device obtains operator information and operator information according to the operator corresponding to the first gesture in the first image. The operator information includes the position of the operator, and the operator information includes the head position and shoulder position of the operator to which the operator belongs.

[0075] The first gesture is a gesture for causing the electronic device to enter a gesture control process. The first gesture may be preset. For example, the first gesture may be set by a technician according to requirements. Optionally, the first gesture may be a static gesture.

[0076] The operator is a user hand whose gesture type in the first image is the first gesture. The operator is the user corresponding to the operator, that is, the operator is the user who makes the first gesture.

[0077] If the first gesture exists in the first image, it means that there is currently an operator who needs to control the electronic device, so the electronic device can obtain the operator information and the operator information according to the operator hand corresponding to the first gesture in the first image, so as to subsequently track the operator's hand through the operator information and the operator information; if the first gesture does not exist in the first image, it means that there is currently no user who wants to control the electronic device, so the electronic device does not need to obtain the operator information and the operator information.

[0078] In some embodiments, after the electronic device acquires the first image in step 101, it can first determine whether the target flag is logically false; if the target flag is logically false, determine whether there is a first gesture in the first image; if there is a first gesture in the first image, execute step 102.

[0079] The target flag is used to indicate whether the gesture control process has started, that is, whether an operator is currently controlling the electronic device.

[0080] If the target flag is logically true, it means that the gesture control process has started, that is, an operator currently controls the electronic device, so there is no need to determine whether the first gesture exists in the first image.

[0081] If the target flag is logically false, it means that the gesture control process has not yet started, that is, no operator has currently controlled the electronic device, so the electronic device needs to determine whether the first gesture exists in the first image to determine whether a user currently wants to control the electronic device. In this case, if the electronic device determines that the first gesture exists in the first image, the target flag can be set to logically true.

[0082] In some embodiments, the operation of the electronic device to determine whether the first gesture exists in the first image may be: the electronic device performs hand recognition on the first image to obtain information of each user hand among one or more user hands in the first image. By way of example, the information of the user hand may include one or more of the position of the user hand (i.e., the position of the user hand in the image), user hand identification, gesture type, user hand size, hand type, etc.; and determines whether the first gesture exists in the first image based on the gesture type of each user hand among all user hands in the first image.

[0083] The user hand identifier is used to identify the user hand in the first image. The user hand identifier is assigned by the electronic device and is unique.

[0084] For example, the position of the user's hand may be the position of each joint point (such as 21 joint points) in the user's hand in the image.

[0085] The gesture type is the type of gesture used to control the electronic device to implement different functions. The gesture type may include a first gesture, a second gesture, etc. The gesture type may be preset.

[0086] The user hand size is the size of the user's hand in the image.

[0087] Hand types can include left and right hands.

[0088] Optionally, when the electronic device performs hand recognition on the first image, the electronic device may input the first image into a gesture recognition model, and the gesture recognition model outputs information about each user's hand in the first image.

[0089] The gesture recognition model is used to recognize the hand and the preset gestures (such as the first gesture and the second gesture). For example, the gesture recognition model can be obtained by training a neural network model.

[0090] In some cases, if there is a user hand whose gesture type is the first gesture among all the user hands in the first image, the user hand is determined as the operating hand; if there is no user hand gesture type of the first gesture among all the user hands in the first image, the electronic device continues to acquire the next frame of image.

[0091] In some cases, if there are multiple user hands in the first image whose gesture type is the first gesture, the electronic device may determine the user hand with the largest size among the multiple user hands as the operating hand.

[0092] In some embodiments, the operation of the electronic device obtaining the operator information and the operator information according to the operator corresponding to the first gesture in the first image may be: the electronic device determines the information of the operator in the first image as the operator information; determines the minimum rectangular area in the first image that contains the operator; expands the range of the minimum rectangular area according to a second ratio to obtain a second area; crops a fourth image located in the second area from the first image; performs human body joint point recognition on the fourth image to obtain the wrist position, head position and shoulder position of each user among one or more users; and determines the head position and shoulder position of the user whose wrist position is the smallest distance between the position of the operator among the one or more users as the operator information.

[0093] The second area is an area that has a high probability of including the operator to whom the operator belongs.

[0094] The second ratio can be preset. The second ratio is greater than 1.

[0095] The fourth image is an image including the operator to whom the operator belongs.

[0096] By performing human joint recognition on the fourth image, the wrist position, head position and shoulder position of each user in the one or more users in the fourth image can be obtained. Since the wrist position of a user is closest to the hand position of the user, the user corresponding to the wrist position with the smallest distance between the wrist position and the position of the operator's hand among the one or more users in the fourth image can be determined as the operator, and the head position and shoulder position of the user are determined as the operator information.

[0097] For example, when the electronic device performs human joint point recognition on the fourth image, the fourth image can be input into a human key point recognition model, and the human key point recognition model outputs the wrist position, head position and shoulder position of each user among one or more users in the fourth image.

[0098] The human body key point recognition model is used to recognize the human body joints of multiple people. For example, the human body key point recognition model can be obtained by training a neural network model.

[0099] After the electronic device obtains the operator information and the operator information, the electronic device may save the operator information and the operator information.

[0100] In some embodiments, the electronic device may start the first timing after acquiring the operator information and the operator information from the operator corresponding to the first gesture in the first image.

[0101] The first timing is used to indicate the start time of correcting the operator information according to the operator information during the operator tracking process.

[0102] In some embodiments, the electronic device may start the second timing after acquiring the operator information and the operator information from the operator corresponding to the first gesture in the first image.

[0103] The second timing is used to indicate the end time of the gesture control process.

[0104] Step 103: The electronic device acquires a second image.

[0105] The second image is a frame of image acquired after the first image. For example, the second image may be a frame of image next to the first image, or may be an image acquired after multiple frames of image after the first image, which is not limited in the present embodiment.

[0106] For example, the second image may be a frame of image captured by a camera of the electronic device.

[0107] Step 104: The electronic device determines the position of each user hand among the one or more user hands in the second image.

[0108] In some embodiments, the electronic device may perform hand recognition on the second image to obtain information about each user's hand in one or more user's hands in the second image. For example, the information about the user's hand may include one or more of the user's hand position, user's hand identifier, gesture type, user's hand size, hand type, etc.

[0109] Optionally, when the electronic device performs hand recognition on the second image, the electronic device may input the second image into a gesture recognition model, and the gesture recognition model outputs information about each user's hand in the second image.

[0110] In some embodiments, after the electronic device determines the position of each of the one or more user hands in the second image, the electronic device may directly execute the following step 105, and may optionally execute the following step 105 when the target flag is logically true. In this case, the operator information is updated according to the operator information in each frame during the tracking of the operator hand, so that the tracking of the operator can be achieved more accurately.

[0111] Alternatively, after the electronic device determines the position of each user's hand in one or more user's hands in the second image, it can compare the timing duration of the first timing with the first preset duration, and optionally compare the timing duration of the first timing with the first preset duration when the target flag is logically true; if the timing duration of the first timing is less than the first preset duration, the operating hand in the second image is determined based on the saved operating hand information and the position of each user's hand in one or more user's hands in the second image, and the operating hand information is updated; if the timing duration of the first timing is greater than or equal to the first preset duration, the following step 105 is executed, and the first timing is reset.

[0112] The first preset duration can be preset. For example, the first preset duration can be set by a technician according to needs.

[0113] In this case, the operator information is corrected every several frames according to the operator information during the tracking process, that is, the operator information of each frame in the continuous multiple frames can be determined according to the saved operator information, and the operator information of the frame obtained after each multiple frame interval can be determined according to the operator information. In this way, continuous tracking and timing correction of the operator can be achieved. In this way, not only the tracking accuracy can be improved, but also the processing pressure can be reduced, the operation speed can be increased, and the sensitivity of use can be improved.

[0114] It should be noted that the saved operator information is the information of the operator in the image before the second image. Since the position change of the operator in a short period of time is usually small, the operator in the second image can be determined based on the saved operator information and the position of each user's hand in one or more user's hands in the second image.

[0115] In some embodiments, the electronic device determines the operation of the operator in the second image based on the saved operator information and the position of each user hand in one or more user hands in the second image. The operation may be: the electronic device determines the user hand with the smallest distance between the one or more user hands in the second image and the position in the operator information as the operator in the second image; or, the electronic device first determines the user hand with the same hand type as the hand type in the operator information from one or more user hands in the second image, and then determines the user hand with the smallest distance between the determined user hand and the position in the operator information as the operator in the second image; or, the electronic device first determines the user hand with the same hand type as the hand type in the operator information from one or more user hands in the second image, and then selects the user hand with the distance between the determined user hand and the position in the operator information less than a preset distance, and then determines the user hand with the smallest distance between the selected user hand and the position in the operator information as the operator in the second image. The preset distance can be set in advance.

[0116] After the electronic device determines the operator in the second image, the stored operator information may be updated to the operator information in the second image.

[0117] Optionally, if the electronic device does not determine the user's hand from the second image, or if the electronic device determines one or more user's hands from the second image but does not determine the operating hand from the one or more user's hands, the timing duration of the second timing can be compared with the second preset duration; if the timing duration of the second timing is greater than or equal to the second preset duration, the gesture control process is terminated; if the timing duration of the second timing is less than the second preset duration, the next frame image is acquired.

[0118] If the second timing duration is less than the second preset duration, it means that the gesture control process has not timed out, so the operator can continue to be tracked. If the second timing duration is greater than or equal to the second preset duration, it means that the gesture control process has timed out, so the gesture control process can be terminated.

[0119] Optionally, if the electronic device determines the operating hand in the second image, it can determine whether the second gesture is recognized based on the operating hand in the second image; if the second gesture is recognized, the control operation corresponding to the second gesture is executed, and then the gesture control process is terminated; if the second gesture is not recognized, the timing duration of the second timing is compared with the second preset duration, if the timing duration of the second timing is greater than or equal to the second preset duration, the gesture control process is terminated, and if the timing duration of the second timing is less than the second preset duration, the next frame image is acquired.

[0120] The second gesture is a gesture for controlling the electronic device. The second gesture can be preset. For example, the second gesture can be set by a technician according to needs. For example, the second gesture can be a static gesture, or the second gesture can be a dynamic gesture (also called a continuous gesture), and the dynamic gesture can be recognized in combination with the posture of the operator in multiple frames of images.

[0121] The control operation corresponding to the second gesture can be pre-set. If the electronic device recognizes the second gesture, it means that the operator is sending a control instruction to the electronic device to try to perform corresponding control. In this case, the electronic device can execute the control operation corresponding to the second gesture to respond to the control instruction.

[0122] In some embodiments, when the electronic device ends the gesture control process, it can reset and stop the first timing and the second timing, delete the saved operator information and operator information, and set the target flag to logical false.

[0123] Step 105: The electronic device determines the target wrist position in the second image according to the operator information.

[0124] The target wrist position is the position of the operator's wrist in the second image.

[0125] Generally, during the process of gesture control, the operator's head and shoulders usually remain still or move only slightly, that is, the position of the operator's head and shoulders changes very little, so in the embodiment of the present application, the operator's hand information can be corrected according to the operator information. That is, the target wrist position can be determined in the second image according to the saved operator information, and then the operator's hand in the second image can be accurately determined based on this information.

[0126] In some embodiments, the operation of step 105 may be: the electronic device crops the second image according to the head position and shoulder position of the operator in the operator information to obtain a third image; and performs wrist detection on the third image to obtain a target wrist position.

[0127] The third image is an image including the operator's wrist.

[0128] Since the operator's wrist position is usually located near the operator's head position and shoulder position, it is very likely that the third image containing the operator's wrist can be obtained by cropping the second image according to the operator's head position and shoulder position. In this case, the wrist position of the operator (i.e., the target wrist position) can be obtained by performing wrist detection on the third image.

[0129] For example, when the electronic device performs wrist detection on the third image, the third image can be input into a human key point recognition model, and the human key point recognition model outputs the wrist position in the third image (i.e., the target wrist position). Optionally, the human key point recognition model can also output the head position and shoulder position in the third image, which are the head position and shoulder position of the operator to whom the operator hand in the second image belongs.

[0130] Optionally, the human body key point recognition model may output a wrist joint point identifier corresponding to the wrist position while outputting the wrist position, and the joint point identifier may indicate whether the wrist is a left wrist or a right wrist.

[0131] In some embodiments, the electronic device crops the second image according to the head position and shoulder position of the operator to obtain the third image by: the electronic device determines the smallest rectangular area in the second image that includes the head position and shoulder position of the operator; expands the range of the smallest rectangular area according to a first ratio to obtain the first area; and crops the third image located in the first area from the second image.

[0132] The first ratio can be preset. The first ratio is greater than 1.

[0133] The first area is an area that has a high probability of including the operator's wrist.

[0134] In some cases, if the electronic device obtains a wrist position after performing wrist detection on the third image, the wrist position can be determined as the target wrist position. If the electronic device obtains multiple wrist positions after performing wrist detection on the third image, the wrist position among the multiple wrist positions that is consistent with the hand type in the stored operating hand information can be determined as the target wrist position.

[0135] For example, the hand type in the operator information is a left hand. The joint point of the left wrist is identified as 20, and the joint point of the right wrist is identified as 21. If the electronic device obtains two wrist positions after performing wrist detection on the third image, the wrist position with the joint point identified as 20 can be determined as the target wrist position.

[0136] In some embodiments, before the electronic device determines the target wrist position in the second image based on the operator information, it can filter the user's hand in the second image to improve the accuracy of subsequent determination of the operator's hand in the second image and reduce the amount of calculation in subsequent determination of the operator's hand in the second image.

[0137] Optionally, the operation of filtering the user hands in the second image by the electronic device may be: the electronic device obtains a target set, the target set including all user hands in the second image; deletes user hands in the target set whose hand types are different from those in the operator hand information; deletes user hands in the target set whose distance from the position in the operator hand information is greater than or equal to a preset distance.

[0138] It can be understood that the target set includes all user hands in the second image means that the target set includes the identifiers of all user hands in the second image.

[0139] Since the operator has a unique hand type, the user's hand in the target set that is different from the hand type in the operator's information can be deleted. Since the position change of the operator in a short period of time is usually small, the user's hand in the target set that is too far away from the position in the operator's information can be deleted. In this way, the user's hand in the target set that is highly likely not to be the operator can be excluded, reducing the interference hands in the subsequent determination of the operator in the second image.

[0140] Step 106: The electronic device determines the operator in the second image according to the position of each of the one or more user hands in the second image and the target wrist position, and updates the operator information and the operator information.

[0141] Generally, the distance between the operator's hand and wrist is very close, so the operator's hand can be more accurately determined from one or more user hands in the second image according to the target wrist position.

[0142] It should be noted that, in the case where there are multiple user hands in the second image, the embodiment of the present application can accurately determine the operator hand from the multiple user hands according to the target wrist position. In this way, even if the operator hand is tracked incorrectly before, it can be corrected in time. That is, the embodiment of the present application can accurately lock and track the operator hand in a multi-person scenario, and then smooth gesture control can be achieved.

[0143] In some embodiments, the operation of step 106 may be: the electronic device determines the user hand with the smallest distance from the target wrist position among all the user hands in the second image as the operating hand in the second image.

[0144] In other embodiments, the operation of step 106 may be: the electronic device determines the operating hand in the second image from the hands of the users in the target set according to the position of each user's hand in all the users' hands in the target set and the target wrist position.

[0145] Since the user hands in the target set are of the same hand type as the operator in the operator information and are relatively close to each other, the electronic device can quickly determine the operator in the second image from the user hands in the target set.

[0146] For example, the electronic device may determine the user hand in the target set whose distance from the target wrist position is the smallest as the operating hand in the second image.

[0147] After the electronic device determines the operator in the second image, the operator information may be updated to the operator information in the second image, and the operator information may be updated to the head position and shoulder position of the operator to which the operator belongs in the second image.

[0148] Optionally, if the electronic device determines the operating hand in the second image, it can determine whether the second gesture is recognized based on the operating hand in the second image; if the second gesture is recognized, the control operation corresponding to the second gesture is executed, and then the gesture control process is terminated; if the second gesture is not recognized, the timing duration of the second timing is compared with the second preset duration, if the timing duration of the second timing is greater than or equal to the second preset duration, the gesture control process is terminated, and if the timing duration of the second timing is less than the second preset duration, the next frame image is acquired.

[0149] For ease of understanding, the following Figure 2 The above-mentioned operator tracking method is exemplified.

[0150] Figure 2 is a flow chart of a method for tracking an operator provided in an embodiment of the present application. Figure 2 The method includes the following steps 201 to 216.

[0151] Step 201: The electronic device acquires a target image.

[0152] The target image may be the first image or the second image mentioned above.

[0153] Step 202: The electronic device performs hand recognition on the target image to obtain a target set.

[0154] The target set includes all user hands in the target image.

[0155] Step 203: The electronic device determines whether the target flag is logically true.

[0156] If the target flag is logically false, it means that the electronic device has not yet entered the gesture control process. In this case, the target image is the first image mentioned above, and step 204 is executed; if the target flag is logically true, it means that the electronic device has entered the gesture control process, that is, it is currently tracking the operating hand. In this case, the target image is the second image mentioned above, and step 207 is executed.

[0157] Step 204: The electronic device determines whether a first gesture exists in the target image.

[0158] If the first gesture exists in the target image, it means that the user wants to perform gesture control, and step 205 is executed; if the first gesture does not exist in the target image, the next frame of image is acquired, that is, step 201 is continued.

[0159] Step 205: The electronic device saves the operator information, and then executes step 206.

[0160] Step 206: the electronic device starts the gesture control process, sets the target flag to logical true, obtains the operator information according to the target image and saves it, and then continues to obtain the next frame of image, that is, continues to execute step 201.

[0161] Step 207 : the electronic device deletes the user hands in the target set that are different from the hand type of the operator in the operator information, and then executes step 208 .

[0162] Step 208 : the electronic device deletes the user's hand in the target set whose distance from the position of the operator in the operator information is greater than or equal to a preset distance, and then executes step 209 .

[0163] Step 209: The electronic device determines whether the target set is empty.

[0164] If the target set is empty, it means that there is no operator in the user's hand in the second image, and then step 215 is executed; if the target set is not empty, it means that there may be an operator in the user's hand in the second image, and then step 210 is executed.

[0165] Step 210: The electronic device determines whether the timing duration of the first timing reaches a first preset duration.

[0166] If the timing duration of the first timing reaches the first preset duration, step 211 is executed; if the timing duration of the first timing does not reach the first preset duration, step 214 is executed.

[0167] Step 211: The electronic device resets the first timing, and then executes step 213.

[0168] Step 212: The electronic device determines the target wrist position in the target image according to the operator information, and then executes step 214.

[0169] Step 213: the electronic device determines the operator from the target set according to the target wrist position, updates the operator information and the operator information, and then executes step 215.

[0170] Step 214 : the electronic device determines the user hand in the target set whose distance from the position of the operator in the operator information is the smallest as the operator, and then executes step 215 .

[0171] Step 215: The electronic device determines whether the timing duration of the second timing reaches a second preset duration.

[0172] If the timing duration of the second timing reaches the second preset duration, step 216 is executed; if the timing duration of the second timing does not reach the second preset duration, the next frame image is acquired, that is, step 201 is executed.

[0173] Step 216: The electronic device ends the gesture control process.

[0174] When the electronic device ends the gesture control process, it can reset and stop the first timing and the second timing, delete the saved operator information and operator information, and set the target flag bit to logical false.

[0175] In an embodiment of the present application, an electronic device acquires a first image; if there is a first gesture in the first image, then the operator information and the operator information are acquired according to the operator corresponding to the first gesture in the first image, the operator information includes the position of the operator, and the operator information includes the head position and the shoulder position of the operator to which the operator belongs. Afterwards, a second image is acquired to determine the position of each user hand in one or more user hands in the second image. Then, the target wrist position in the second image is determined according to the operator information. Since the operator's head and shoulders usually do not undergo a large displacement in the gesture control process, the target wrist position can be accurately determined according to the operator information. Since the distance between the operator's wrist and hand is very close, the operator in the second image can be more accurately determined according to the position of each user hand in one or more user hands in the second image and the target wrist position, and the operator information and the operator information are updated accordingly. In this way, accurate tracking of the operator can be achieved.

[0176] Figure 3 Schematic diagram of the structure of an operator tracking device provided in an embodiment of the present application. The device can be implemented by software, hardware or a combination of both to form part or all of a computer device, and the computer device can be as follows Figure 4 Computer equipment shown. Figure 3 The device includes: a first acquisition module 301, a second acquisition module 302, a third acquisition module 303, a first determination module 304, a second determination module 305, and a third determination module 306.

[0177] A first acquisition module 301, used to acquire a first image;

[0178] A second acquisition module 302 is used to acquire, if the first gesture exists in the first image, information about the operator and information about the operator according to the operator corresponding to the first gesture in the first image, wherein the information about the operator includes the position of the operator, and the information about the operator includes the position of the head and the shoulder of the operator to whom the operator belongs;

[0179] A third acquisition module 303 is used to acquire a second image, where the second image is a frame of image acquired after the first image;

[0180] A first determination module 304, configured to determine a position of each user hand in one or more user hands in the second image;

[0181] A second determination module 305 is used to determine the target wrist position in the second image according to the operator information;

[0182] The third determination module 306 is used to determine the operator in the second image according to the position of each of the one or more user hands in the second image and the target wrist position, and update the operator information and the operator information.

[0183] Optionally, the device further comprises:

[0184] a fourth determination module, configured to determine whether a first gesture exists in the first image if the target flag is logically false;

[0185] The device also includes:

[0186] The setting module is used to set the target flag bit to logical true if the first gesture exists in the first image.

[0187] Optionally, the second determining module 305 is used to:

[0188] If the target flag is logically true, the target wrist position is determined based on the operator information.

[0189] Optionally, the device further comprises:

[0190] A first timing module, used to start a first timing;

[0191] The second determining module 305 is used for:

[0192] If the timing duration of the first timing is greater than or equal to the first preset duration, the target wrist position is determined according to the operator information, and the first timing is reset.

[0193] Optionally, the device further comprises:

[0194] The fifth determination module is used to determine the operator in the second image and update the operator information based on the operator information and the position of each user's hand in one or more user's hands in the second image if the timing duration of the first timing is less than the first preset duration.

[0195] Optionally, the operator information also includes the operator's hand type, and the device further includes:

[0196] A fourth acquisition module, used to acquire a target set, where the target set includes all user hands in the second image;

[0197] A first deletion module is used to delete the user's hand of a different type from the hand in the operator's hand information in the target set;

[0198] A second deleting module is used to delete a user hand in the target set whose distance from the position in the operator hand information is greater than or equal to a preset distance;

[0199] The third determination module 306 is used for:

[0200] According to the position of each user's hand among all the user's hands in the target set and the target wrist position, the operating hand in the second image is determined from the user's hands in the target set.

[0201] Optionally, the second determining module 305 is used to:

[0202] The second image is cropped according to the head position and shoulder position of the operator to obtain a third image;

[0203] Perform wrist detection on the third image to obtain a target wrist position.

[0204] Optionally, the second determining module 305 is used to:

[0205] Determine the smallest rectangular area in the second image that includes the head position and shoulder position of the operator;

[0206] Expanding the range of the minimum rectangular area according to a first ratio to obtain a first area;

[0207] A third image located within the first area is cropped from the second image.

[0208] Optionally, the third determining module 306 is used to:

[0209] The user hand of the one or more user hands in the second image whose distance to the target wrist position is the smallest is determined as the operating hand in the second image.

[0210] Optionally, the device further comprises:

[0211] A second timing module, used to start a second timing;

[0212] The device also includes:

[0213] a sixth determination module, configured to determine whether a second gesture is recognized according to the operator in the second image;

[0214] The ending module is used to end the gesture control process if the second gesture is not recognized and the timing duration of the second timing is greater than or equal to the second preset duration.

[0215] Optionally, the device further comprises:

[0216] The control module is configured to execute a control operation corresponding to the second gesture if the second gesture is recognized.

[0217] In an embodiment of the present application, the device acquires a first image; if there is a first gesture in the first image, the operator information and the operator information are acquired according to the operator corresponding to the first gesture in the first image, the operator information includes the position of the operator, and the operator information includes the head position and shoulder position of the operator to which the operator belongs. Afterwards, the second image is acquired to determine the position of each user hand in one or more user hands in the second image. Then, the target wrist position in the second image is determined according to the operator information. Since the operator's head and shoulders usually do not undergo a large displacement in the gesture control process, the target wrist position can be accurately determined according to the operator information. And since the distance between the operator's wrist and hand is very close, the operator in the second image can be more accurately determined according to the position of each user hand in one or more user hands in the second image and the target wrist position, and the operator information and the operator information are updated accordingly. In this way, accurate tracking of the operator can be achieved.

[0218] It should be noted that: the operator tracking device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to illustrate when tracking the operator. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0219] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0220] The operator tracking device and the operator tracking method provided in the above embodiments belong to the same concept. The specific working process and technical effects brought about by the units and modules in the above embodiments can be found in the method embodiment part and will not be repeated here.

[0221] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 4 As shown, the computer device 4 includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, the steps of the operating hand tracking method in the above embodiment are implemented.

[0222] The computer device 4 may be a general-purpose computer device or a dedicated computer device. In a specific implementation, the computer device 4 may be a desktop computer, a portable computer, a network server, a PDA, a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of the computer device 4. Those skilled in the art will understand that Figure 4 It is only an example of the computer device 4 and does not constitute a limitation on the computer device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, etc.

[0223] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0224] In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the computer device 4. The memory 41 is used to store an operating system, an application program, a boot loader, data, and other programs. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0225] An embodiment of the present application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and when the processor executes the computer program, the steps in any of the above-mentioned method embodiments are implemented.

[0226] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0227] An embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the steps in the above-mentioned method embodiments.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above method embodiments, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device. The computer-readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, it can be a non-transient storage medium.

[0229] It should be understood that all or part of the steps to implement the above embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-mentioned computer readable storage medium.

[0230] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0231] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0232] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0233] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0235] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for tracking an operator, characterized in that: The method comprises: acquiring a first image; If the first gesture exists in the first image, acquiring operator information and operator information according to the operator corresponding to the first gesture in the first image, the operator information including the position of the operator, and the operator information including the head position and shoulder position of the operator to which the operator belongs; Acquire a second image, where the second image is a frame of image acquired after the first image; determining a position of each of the one or more user hands in the second image; determining a target wrist position in the second image according to the operator information; According to the position of each of the one or more user hands in the second image and the target wrist position, the operator in the second image is determined, and the operator information and the operator person information are updated.

2. The method according to claim 1, characterized in that Before acquiring the operator information and the operator information according to the operator corresponding to the first gesture in the first image, the method further includes: If the target flag is logically false, determining whether the first gesture exists in the first image; The method further comprises: If the first gesture exists in the first image, the target flag is set to logical true.

3. The method according to claim 2, characterized in that The step of determining the target wrist position in the second image according to the operator information includes: If the target flag is logically true, the target wrist position is determined according to the operator information.

4. The method according to claim 1, characterized in that After acquiring the operator information and the operator information according to the operator corresponding to the first gesture in the first image, the method further includes: Start the first count; The step of determining the target wrist position in the second image according to the operator information includes: If the timing duration of the first timing is greater than or equal to the first preset duration, determining the target wrist position according to the operator information, and resetting the first timing; After determining the position of each of the one or more user hands in the second image, the method further includes: If the timing duration of the first timing is less than the first preset duration, the operator in the second image is determined according to the operator information and the position of each user hand of one or more user hands in the second image, and the operator information is updated.

5. The method according to claim 1, characterized in that The operator information also includes the type of the operator's hand. Before determining the target wrist position in the second image according to the operator information, the method further includes: Acquire a target set, where the target set includes all user hands in the second image; Deleting the user hands in the target set that are different in type from the hand in the operator hand information; Deleting a user hand from the target set whose distance from the position in the operator hand information is greater than or equal to a preset distance; The determining the operating hand in the second image according to the position of each of the one or more user hands in the second image and the target wrist position includes: According to the position of each user's hand among all the user's hands in the target set and the target wrist position, the operating hand in the second image is determined from the user's hands in the target set.

6. The method according to claim 1, characterized in that The step of determining the target wrist position in the second image according to the operator information includes: cropping the second image according to the head position and shoulder position of the operator to obtain a third image; Perform wrist detection on the third image to obtain the target wrist position.

7. The method according to claim 6, characterized in that The step of cropping the second image according to the head position and shoulder position of the operator to obtain a third image includes: Determine a minimum rectangular area in the second image that includes the head position and shoulder position of the operator; Expanding the minimum rectangular area according to a first ratio to obtain a first area; The third image located within the first area is cropped from the second image.

8. The method according to claim 1, characterized in that The determining the operating hand in the second image according to the position of each of the one or more user hands in the second image and the target wrist position includes: The user hand whose distance from the target wrist position is the smallest among the one or more user hands in the second image is determined as the operating hand in the second image.

9. The method according to any one of claims 1 to 8, characterized in that: After obtaining the operator information, the method further includes: Start the second timer; After determining the operator in the second image, the method further includes: determining whether a second gesture is recognized according to the operator in the second image; If the second gesture is not recognized, and the timing duration of the second timing is greater than or equal to the second preset duration, then the gesture control process is terminated; After determining whether the second gesture is recognized according to the operator in the second image, the method further includes: If the second gesture is recognized, the control operation corresponding to the second gesture is executed.

10. An operator tracking device, characterized in that: The device comprises: A first acquisition module, used for acquiring a first image; a second acquisition module, configured to acquire, if a first gesture exists in the first image, operator information and operator person information according to an operator corresponding to the first gesture in the first image, wherein the operator information includes a position of the operator, and the operator person information includes a head position and a shoulder position of an operator to which the operator belongs; A third acquisition module is used to acquire a second image, where the second image is a frame of image acquired after the first image; a first determination module, configured to determine a position of each of the one or more user hands in the second image; A second determination module, configured to determine a target wrist position in the second image according to the operator information; The third determination module is used to determine the operator in the second image according to the position of each user's hand of one or more users in the second image and the target wrist position, and update the operator information and the operator information.

11. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by the processor.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.