Gesture tracking method and apparatus, and computer device and readable storage medium

Abstract

Provided in the present disclosure are a gesture tracking method and apparatus, and a computer device and a readable storage medium. The method comprises: acquiring a vehicle cabin video data stream, performing image capturing on the vehicle cabin video data stream to obtain several image frames; triggering hand detection on the image frames at preset time intervals, so as to obtain hand position information of each triggered image frame; for each triggered image frame, determining hand position information of each non-triggered image frame on the basis of the hand position information of the triggered image frame; matching hand position information of hands on the basis of hand position information of every two consecutive image frames to obtain a matching result of the hand position information of the hands; and determining a dynamic gesture in the vehicle cabin video data stream on the basis of the matching result of the hand position information of the hands in every two consecutive image frames. By means of the method, the computing power required for gesture tracking is reduced, the requirements for the computing capability of a device are lowered, and the limitations are reduced, thereby ensuring that gesture tracking can be normally performed by means of each device.

Description

Gesture tracking method and device, computer device and readable storage medium

[0001] Cross-reference to related applications

[0002] The present disclosure claims priority to the Chinese patent application No. 202411118583.8, filed on August 15, 2024, and entitled “Gesture tracking method and device, computer device and readable storage medium”, the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the field of computer technology, in particular, to a gesture tracking method and device, computer device and readable storage medium. BACKGROUND

[0004] With the rapid development of the automotive industry, intelligent cockpit technology has become a key factor in enhancing the competitiveness of modern cars. Intelligent cockpit is not just a driving space, but also an advanced human-machine interaction platform that integrates various sensors and interactive interfaces. On this platform, multi-hand tracking and matching technology is particularly important because it directly affects the convenience and safety of user operations. Multi-hand tracking technology allows the system to monitor and recognize the hands of the driver and passengers in real time through cameras. The hand timing action information maintained is converted into corresponding commands such as adjusting the volume, switching the play list, answering the phone, etc. This non-physical contact interaction greatly enriches the experience of drivers and passengers, while reducing distractions during driving and enhancing driving safety.

[0005] In the prior art, when tracking gestures, the gesture detection processing scheme used is to use a deep learning model to detect and process each frame of the video stream to output hand gesture box information, and then track the gesture based on the gesture box information output by the model. However, it is found in research that since the deep learning model needs to be called to detect and process each frame of image, a large amount of computing power is required, making the existing gesture detection processing scheme have high requirements for the computing power of the device, great limitations, and difficult to apply to devices with weak computing performance. For some cockpit systems and devices with weak computing performance, it is impossible to provide enough computing resources for multiple model runs, so it is impossible to achieve gesture tracking through the existing gesture detection processing.

[0006] SUMMARY

[0007] Therefore, the present disclosure aims to provide a gesture tracking method, device, computer equipment and readable storage medium to reduce the computing power required for gesture tracking, reduce the requirement for device computing capability for gesture tracking, reduce the limitations of gesture tracking, and ensure that gesture tracking can be normally performed through various devices.

[0008] In a first aspect, the embodiments of the present disclosure provide a gesture tracking method, which comprises:

[0009] Obtaining a vehicle cabin video data stream, and performing image capture on the vehicle cabin video data stream to obtain a plurality of image frames;

[0010] Triggering hand detection on the image frames every preset time interval to obtain hand position information of each triggered image frame, wherein the hand position information comprises coordinate information of a hand bounding box of each hand;

[0011] For each triggered image frame, determining hand position information of each non-triggered image frame according to the hand position information of the triggered image frame, wherein the non-triggered image frame is a video frame after the triggered image frame and before a next triggered image frame;

[0012] Matching the hand position information of each hand based on the hand position information of each two consecutive image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure;

[0013] Determining a dynamic gesture in the vehicle cabin video data stream based on the matching result of the hand position information of each hand in each two consecutive image frames.

[0014] Optionally, after determining the hand position information of each non-triggered image frame according to the hand position information of the triggered image frame for each triggered image frame, the method further comprises:

[0015] Determining hand key points of each hand and confidence of each hand key point in each non-triggered image frame according to the hand position information of each non-triggered image frame;

[0016] For each non-triggered image frame, determining whether an average value of the confidence of each hand key point in the non-triggered image frame exceeds a preset threshold value;

[0017] If the average value of the confidence of each hand key point in the non-triggered image frame does not exceed the preset threshold value, triggering hand detection on the non-triggered image frame to obtain new hand position information of the non-triggered image frame;

[0018] Updating the hand position information of the non-triggered image frame determined according to the hand position information of the triggered image frame by using the new hand position information of the non-triggered image frame.

[0019] Optionally, the hand detection on the image frame is triggered every preset time length, and hand position information of each triggered image frame is obtained, including:

[0020] Down-sampling processing is performed on each triggered image.

[0021] The hand position information of each triggered image frame is obtained by inputting the down-sampled triggered image into the trained hand detection model.

[0022] Optionally, the hand position information of each non-triggered image frame is determined according to the hand position information of the triggered image frame, including:

[0023] The coordinate information of the hand bounding box of each hand in the triggered image frame is expanded by a preset multiple to obtain the coordinate information of the hand bounding box of each hand in each non-triggered image frame, so as to determine the hand position information of each non-triggered image frame.

[0024] Optionally, the matching result of the hand position information of each hand is obtained by matching the hand position information of each hand based on the hand position information of each two continuous image frames, including:

[0025] The hand key point position information of each hand in each image frame is determined based on the hand position information of each image frame.

[0026] For each two continuous image frames, a weighted bipartite graph is constructed according to the hand key point position information of each hand and the hand feature information of each hand in the two continuous image frames.

[0027] According to the weighted bipartite graph, the hand position information of each hand in the two continuous images is matched by using the KM algorithm to obtain the matching result of the hand position information of each hand.

[0028] Optionally, the hand key point position of each hand in each image frame is determined based on the hand position information of each image frame, including:

[0029] Each hand image is obtained by cutting out the image region indicated by the hand position information of each image frame.

[0030] The hand key point position of each hand is obtained by inputting each hand image into the trained hand feature point regression model.

[0031] Optionally, the dynamic hand gesture in the vehicle cabin video data stream is determined based on the matching result of the hand position information of each hand in each two continuous image frames, including:

[0032] For each hand, when a matching result of hand position information of the hand in each two continuous image frames is a matching failure, the hand in a later image frame of each two continuous image frames is taken as a start of a new hand action sequence to construct a hand action sequence;

[0033] When the matching result of the hand position information of the hand in each two continuous image frames is a matching success, the hand in each two continuous image frames is taken as a continuous part of a same hand action sequence to construct a hand action sequence.

[0034] According to the hand action sequence constructed, a dynamic hand gesture in the vehicle cabin video data stream is generated.

[0035] In a second aspect, the embodiments of the present disclosure provide a dynamic hand gesture tracking device, and the device comprises:

[0036] An image frame acquisition module is configured to acquire a vehicle cabin video data stream, and perform image capturing on the vehicle cabin video data stream to obtain a plurality of image frames.

[0037] A first hand position information determination module is configured to trigger hand detection on the image frames every preset time length to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand bounding box containing each hand.

[0038] A second hand position information determination module is configured to, for each triggered image frame, determine hand position information of each non-triggered image frame according to the hand position information of the triggered image frame, wherein the non-triggered image frame is a video frame after the triggered image frame and before a next triggered image frame.

[0039] A hand position information matching module is configured to match hand position information of each hand based on hand position information of each two continuous image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure.

[0040] A dynamic hand gesture tracking module is configured to determine a dynamic hand gesture in the vehicle cabin video data stream based on the matching result of the hand position information of each hand in each two continuous image frames.

[0041] Optionally, the device further comprises a position information updating module, which is configured to:

[0042] After determining the hand position information of each non-triggered image frame according to the hand position information of the triggered image frame for each triggered image frame, determine hand key points of each hand in each non-triggered image frame and a confidence of each hand key point according to the hand position information of each non-triggered image frame.

[0043] For each non-trigger image frame, it is judged whether the average of the confidence of each hand key point in the non-trigger image frame exceeds a preset threshold value;

[0044] If the average of the confidence of each hand key point in the non-trigger image frame does not exceed the preset threshold value, hand detection of the non-trigger image frame is triggered to obtain new hand position information of the non-trigger image frame;

[0045] The hand position information of the non-trigger image frame determined according to the hand position information of the trigger image frame is updated by using the new hand position information of the non-trigger image frame.

[0046] Optionally, the hand detection of the image frame is triggered every preset time length to obtain the hand position information of each trigger image frame, comprising:

[0047] Each trigger image is down-sampled;

[0048] Each trigger image after down-sampling is input into a trained hand detection model to obtain the hand position information of each trigger image frame.

[0049] Optionally, the hand position information of each non-trigger image frame is determined according to the hand position information of the trigger image frame, comprising:

[0050] The coordinate information of the hand bounding box of each hand in the trigger image frame is expanded by a preset multiple to obtain the coordinate information of the hand bounding box of each non-trigger image frame, so as to determine the hand position information of each non-trigger image frame.

[0051] Optionally, the hand position information of each hand is matched based on the hand position information of each two consecutive image frames to obtain the matching result of the hand position information of each hand, comprising:

[0052] The hand key point position information of each hand in each image frame is determined based on the hand position information of each image frame;

[0053] For each two consecutive image frames, a weighted bipartite graph is constructed according to the hand key point position information of each hand in the two consecutive image frames and the chirality information of each hand;

[0054] According to the weighted bipartite graph, the hand position information of each hand is matched by using the KM algorithm to obtain the matching result of the hand position information of each hand.

[0055] Optionally, the hand key point position of each hand in each image frame is determined based on the hand position information of each image frame, comprising:

[0056] Each hand image is obtained by cutting out the image region indicated by the hand position information of each image frame;

[0057] inputting each hand image into the trained hand feature point regression model to obtain hand key point positions of each hand.

[0058] Optionally, determining the dynamic hand gesture in the vehicle cabin video data stream based on the matching result of the hand position information of each hand in each two continuous image frames comprises:

[0059] For each hand, when the matching result of the hand position information of the hand in each two continuous image frames is a matching failure, constructing a hand action sequence by taking the hand in the latter image frame in each two continuous image frames as the start of a new hand action sequence.

[0060] When the matching result of the hand position information of the hand in each two continuous image frames is a matching success, constructing a hand action sequence by taking the hand in each two continuous image frames as a continuous part of the same hand action sequence.

[0061] Generating the dynamic hand gesture in the vehicle cabin video data stream according to the constructed hand action sequence.

[0062] In a third aspect, the embodiments of the present disclosure provide a computer device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the gesture tracking method in any one of the optional implementation manners of the first aspect.

[0063] In a fourth aspect, the embodiments of the present disclosure provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of the gesture tracking method in any one of the optional implementation manners of the first aspect.

[0064] The technical solutions provided by the present disclosure include but are not limited to the following beneficial effects:

[0065] The present disclosure can reduce the number of image frames that need to be detected, reduce the number of times of hand detection, and only perform hand detection on the trigger image frames, thereby reducing the computing power required for gesture tracking.

[0066] Then, for each trigger image frame, the hand position information of each non-trigger image frame is determined according to the hand position information of the trigger image frame, so that the hand position information of the non-trigger image frame can be determined without the need for hand detection on other non-trigger image frames that do not perform hand detection, thereby providing complete hand position information of each image frame for gesture detection.

[0067] Finally, the hand position information of each hand is matched based on the hand position information of each two continuous image frames to obtain a matching result of the hand position information of each hand, and a dynamic gesture in the vehicle cabin video data stream is determined based on the matching result of the hand position information of each hand in each two continuous image frames, so that the dynamic gesture in the vehicle cabin video can be tracked, identified and detected according to the matching result of the hand position information in each image frame, thereby realizing low-power gesture tracking in the vehicle cabin.

[0068] By using the above scheme, the number of times of hand detection triggering is limited, and the model is only called to perform hand detection on the image frame that triggers hand detection, and the model is not used to perform hand detection on the image frame that does not trigger hand detection, thereby greatly reducing the power consumption during gesture tracking. Meanwhile, the hand detection result of the image frame that does not trigger hand detection is determined based on the hand detection result of the image frame that triggers hand detection, so that complete and accurate hand data can be provided for subsequent hand matching, the missed detection of the hand can be effectively reduced, and the accuracy of hand detection can be improved. In the gesture tracking process, the non-trigger image frame with inaccurate hand detection result is also re-detected, so that the probability of hand tracking failure caused by too fast hand movement can be reduced, and the accuracy of the hand detection result can be improved. The bipartite graph is established by a specific weight value after triggering detection, and the optimal matching is solved by the KM algorithm, so that the timing information of the hand can be accurately maintained, thereby reducing the power consumption during gesture tracking, reducing the requirement for the computing capacity of the device during gesture tracking, reducing the limitation of gesture tracking, and ensuring accurate gesture tracking by each device.

[0069] In order to make the above-mentioned purposes, features and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0071] FIG. 1 shows a flowchart of a gesture tracking method provided by an embodiment of the present disclosure;

[0072] FIG. 2 shows a flowchart of a position information updating method according to an embodiment of the present disclosure;

[0073] FIG. 3 shows a flowchart of a position information determining method according to an embodiment of the present disclosure;

[0074] FIG. 4 shows a flowchart of a hand position information matching method according to an embodiment of the present disclosure;

[0075] FIG. 5 shows a flowchart of a hand key point positioning method according to an embodiment of the present disclosure;

[0076] FIG. 6 shows a flowchart of a dynamic gesture determining method according to an embodiment of the present disclosure;

[0077] FIG. 7 shows a flowchart of a gesture tracking method in an intelligent cockpit according to an embodiment of the present disclosure;

[0078] FIG. 8 shows a structural schematic diagram of a dynamic gesture tracking device according to an embodiment of the present disclosure;

[0079] FIG. 9 shows a structural schematic diagram of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure and not all embodiments. The components of the embodiments of the present disclosure described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0081] Embodiment One

[0082] In order to facilitate the understanding of the present disclosure, the following describes the embodiment one of the present disclosure in detail with reference to the flowchart of the gesture tracking method provided by the embodiment one of the present disclosure shown in FIG. 1.

[0083] Referring to FIG. 1, FIG. 1 shows a flowchart of a gesture tracking method according to an embodiment of the present disclosure, wherein the method comprises steps S101-S104:

[0084] S101: Obtain a vehicle cabin video data stream, and perform image capture on the vehicle cabin video data stream to obtain a plurality of image frames.

[0085] Specifically, after a vehicle cabin occupant enters the vehicle cabin, the vehicle cabin video data stream is collected by a camera, and hand detection is performed on each image frame.

[0086] S102: Trigger hand detection on the image frames every preset time length to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand bounding box containing each hand.

[0087] Specifically, hand detection on the image frames is triggered every 500 ms. The image frame on which hand detection is triggered is referred to as a triggered image frame. For each triggered image frame, hand detection is performed on the triggered image frame by using a Nanodet-Plus hand detection model to obtain hand position information of the triggered image frame, that is, coordinate information of a hand bounding box containing each hand in the triggered image frame, that is, horizontal and vertical coordinates of the upper left corner and the lower right corner of the hand bounding box.

[0088] S103: For each triggered image frame, determine hand position information of each non-triggered image frame according to the hand position information of the triggered image frame, wherein the non-triggered image frame is a video frame after the triggered image frame and before a next triggered image frame.

[0089] Specifically, since hand detection is performed every preset time length, there are a plurality of non-triggered image frames between each two hand detections, that is, between each two adjacent triggered image frames. For these non-triggered image frames on which hand detection is not performed, the hand position information of the non-triggered image frame is determined based on the hand position information of a historically nearest triggered image frame of the non-triggered image frame.

[0090] S104: Match hand position information of each hand based on hand position information of each two consecutive image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure.

[0091] Specifically, for each image frame, the hand position information of the image frame and the hand position information of a previous image frame of the image frame are matched by optimal matching of a weighted bipartite graph to obtain an optimal matching result.

[0092] S105: Determine a dynamic gesture in the vehicle cabin video data stream based on the matching result of the hand position information of each hand in each two consecutive image frames.

[0093] Specifically, for each image frame, if the hand position information of the image frame fails to match the hand position information of the previous image frame of the image frame (i.e., the matching fails), the hand that fails to match in the image frame is assigned a new hand identifier, indicating the start of a new hand action sequence. If the hand position information of the image frame matches the hand position information of the previous image frame of the image frame, the two hands that match are considered to be continuous parts of the same hand action sequence, and the hand of the image frame will continue to keep the hand identifier of the previous image frame of the image frame. The start of a new hand action sequence is taken as the start of a dynamic gesture, and the continuous parts of each hand action sequence are taken as components after the start of the dynamic gesture, to construct the dynamic gesture.

[0094] In an optional embodiment, referring to FIG. 2, FIG. 2 shows a flowchart of a position information updating method provided by Embodiment One of the present disclosure, wherein after the hand position information of each non-trigger image frame is determined according to the hand position information of the trigger image frame, the method further includes steps S201-S204:

[0095] S201: Determine the hand key points of each hand in each non-trigger image frame and the confidence of each hand key point according to the hand position information of each non-trigger image frame.

[0096] Specifically, the hand images containing each hand are cut out according to the hand position information of each non-trigger image frame, and then each hand image is input into the hand 21-point feature point regression model to obtain the specific positions of 21 hand skeleton points in each non-trigger image and the confidence of each point position. The hand 21-point feature point regression model uses a lightweight QaRepVgg feature extraction network and is trained using two loss functions. RLEloss is used for the positioning task of hand key points, and outputs the coordinates and confidence of each hand key point. CrossEntropyLoss is used to determine the chirality of the hand, and outputs the left and right nature of the hand, i.e., left hand or right hand.

[0097] S202: For each non-trigger image frame, determine whether the average value of the confidence of each hand key point in the non-trigger image frame exceeds a preset threshold.

[0098] Specifically, for each non-trigger image frame, it is determined whether the average value of the confidence of each hand key point in the non-trigger image frame exceeds a preset threshold, for example, 0.2.

[0099] S203: If the average value of the confidence of each hand key point in the non-trigger image frame does not exceed the preset threshold, trigger hand detection of the non-trigger image frame to obtain new hand position information of the non-trigger image frame.

[0100] Specifically, if the average value of the confidence of the hand key points of each hand in the non-trigger image frame does not exceed the preset threshold, it is considered that the key point detection of the non-trigger image frame is inaccurate, and the hand detection of the non-trigger image frame is triggered, and the Nanodet-Plus hand detection model is used to perform hand detection on the non-trigger image frame to obtain accurate new hand position information.

[0101] S204: Update the hand position information of the non-trigger image frame determined according to the hand position information of the trigger image frame using the new hand position information of the non-trigger image frame.

[0102] Specifically, the hand position information of the non-trigger image frame obtained in step S103 is updated using the new hand position information of the non-trigger image frame, so that step S104 performs hand matching according to the new hand position information of the non-trigger image frame.

[0103] If the average value of the confidence of the hand key points of each hand in the non-trigger image frame exceeds the preset threshold, it is considered that the key point detection of the non-trigger image frame is accurate and can be used, and the hand detection of the non-trigger image frame does not need to be triggered, and the hand matching is directly performed according to the hand position information of the non-trigger image frame obtained in step S103.

[0104] In an optional embodiment, referring to FIG. 3, FIG. 3 shows a flowchart of a position information determination method provided by an embodiment of the present disclosure, wherein the hand detection of the image frame is triggered every preset time length to obtain the hand position information of each trigger image frame, including steps S301-S302:

[0105] S301: Perform down-sampling processing on each trigger image.

[0106] Specifically, the down-sampling processing is performed on each trigger image to reduce the image size.

[0107] S302: Input each trigger image after the down-sampling processing into the trained hand detection model to obtain the hand position information of each trigger image frame.

[0108] Specifically, the hand detection model is a Nanodet-Plus hand detection model.

[0109] In an optional embodiment, the hand position information of each non-trigger image frame is determined according to the hand position information of the trigger image frame, including:

[0110] The coordinate information of the hand bounding box of each hand in the trigger image frame is expanded by a preset multiple to obtain the coordinate information of the hand bounding box of each non-trigger image frame, so as to determine the hand position information of each non-trigger image frame.

[0111] Specifically, for each non-trigger image frame, the hand bounding box of the history nearest trigger image frame of the non-trigger image frame is expanded to obtain the hand bounding box of the non-trigger image frame as the hand position information of the non-trigger image frame. The history nearest trigger image frame is the trigger image frame closest to the non-trigger image.

[0112] In an optional embodiment, referring to FIG. 4, FIG. 4 shows a flowchart of a hand position information matching method provided by an embodiment of the present disclosure, wherein the matching of the hand position information of each hand based on the hand position information of each two continuous image frames obtains the matching result of the hand position information of each hand, including steps S401-S403:

[0113] S401: determining the hand key point position information of each hand in each image frame based on the hand position information of each image frame.

[0114] Specifically, the hand image containing each hand is cut out according to the hand bounding box information of each image frame, and then each hand image is input into the hand 21-point feature point regression model to obtain the position information of 21 hand key points in each image frame and the confidence of each hand key point.

[0115] S402: for each two continuous image frames, a weighted bipartite graph is constructed according to the hand key point position information of each hand and the hand information of each hand in the two continuous image frames.

[0116] Specifically, the construction method of the weighted bipartite graph is as follows:

[0117] For each two continuous image frames, the hand set detected in the latter image frame of the two continuous image frames is denoted as A, the hand set in the former image frame of the two continuous image frames, i.e., the hand set maintained in the previous preset length (optionally 3s) window is denoted as B, and A and B are taken as the left and right parts of the bipartite graph respectively. For each element a in A and each element b in B, the distance of each a and each b is calculated according to the hand key point position information of each hand, if the distance of a and b is less than a preset threshold D, then a and b are connected, and the edge weight W(a, b) between a and b is: W(a, b) = 1 / (x*dist(a, b)+K), wherein if a and b are both left hands or right hands, x is a first preset parameter X0, otherwise x is a second preset parameter X1 (wherein 0X0X1), dist(a, b) represents the Euclidean distance of the wrist key point coordinates of a and b, and k is a preset fixed parameter (positive number).

[0118] S403: According to the weighted bipartite graph, the hand position information of each hand in the two continuous images is matched by using the KM algorithm to obtain a matching result of the hand position information of each hand.

[0119] Specifically, based on the weighted bipartite graph constructed by the above method, the hand position information of each hand in the two continuous images is matched by using the KM algorithm, and the matching result includes matching success and matching failure. When the hand position information of a certain hand in the two continuous images matches each other, it means that the matching result of the hand is matching success; when the hand position information of a certain hand in the two continuous images does not match, it means that the matching result of the hand is matching failure.

[0120] In an optional embodiment, referring to FIG. 5, FIG. 5 shows a flowchart of a hand key point positioning method provided by the first embodiment of the present disclosure, wherein the hand key point positions of each hand in each image frame are determined based on the hand position information of each image frame, including steps S501-S502:

[0121] S501: The image region indicated by the hand position information of each image frame is cut out to obtain each hand image.

[0122] S502: Each hand image is input into the trained hand feature point regression model to obtain the hand key point positions of each hand.

[0123] Specifically, the hand feature point regression model can also output the confidence of each hand key point and the chirality of each hand.

[0124] In an optional embodiment, referring to FIG. 6, FIG. 6 shows a flowchart of a dynamic hand gesture determination method provided by the first embodiment of the present disclosure, wherein the dynamic hand gesture in the vehicle cabin video data stream is determined based on the matching result of the hand position information of each hand in each two continuous image frames, including steps S601-S603:

[0125] S601: For each hand, when the matching result of the hand position information of the hand in each two continuous image frames is matching failure, the hand in the latter image frame in each two continuous image frames is taken as the start of a new hand action sequence to construct a hand action sequence.

[0126] Specifically, a new hand identifier is assigned to the hand.

[0127] S602: When the matching result of the hand position information of the hand in each two continuous image frames is matching success, the hand in each two continuous image frames is taken as a continuous part of the same hand action sequence to construct a hand action sequence.

[0128] Specifically, the hand is assigned with the same hand identity as the previous image frame, i.e. the hand identity is continued from the previous image frame.

[0129] S603: generating a dynamic hand gesture in the vehicle cabin video data stream according to the constructed hand action sequence.

[0130] Specifically, hands with the same hand identity are regarded as components of the same hand action sequence, the hand identities in different hand action sequences are different, the hand identities in the same hand action sequence are the same, and for each hand action sequence, the dynamic hand gesture of the hand in the hand action sequence is obtained through tracking and identification of the hand.

[0131] When the gesture tracking method provided by the present disclosure is applied to tracking and identifying the gestures of the vehicle cabin passengers in the intelligent cabin, referring to FIG. 7, FIG. 7 shows a flowchart of a gesture tracking method in an intelligent cabin provided by Embodiment One of the present disclosure, which specifically includes the following steps: acquiring current frame video stream data in real time, detecting whether no hand is detected in the previous frame; if not, triggering hand detection; if so, determining whether the time since the last hand detection is greater than 500 ms, if so, triggering hand detection, if not, expanding the bounding box of the 21 hand key points detected in the previous frame to obtain the bounding box of the current frame, and then calculating the key points of the hand and the confidence of each key point in the current frame based on the bounding box of the current frame; determining whether the average value of the confidence of each key point is less than a preset threshold, if so, triggering hand detection, if not, ending hand tracking. When hand detection is triggered, the key points of the hand, the confidence of each hand key point and the left and right hand classification information of each hand are determined, then an adjacency matrix is established by using the bounding box of the current frame and the information of the hand that appeared last time within 3s to form a weighted bipartite graph, and the hand matched to the bounding box of the current frame is solved by using the KM algorithm, thereby completing and ending hand tracking.

[0132] Embodiment Two

[0133] Referring to FIG. 8, FIG. 8 shows a structural schematic diagram of a dynamic gesture tracking device provided by Embodiment Two of the present disclosure, wherein the device includes:

[0134] An image frame acquisition module 801 is configured to acquire a vehicle cabin video data stream, and capture images from the vehicle cabin video data stream to obtain a plurality of image frames.

[0135] A first hand position information determination module 802 is configured to trigger hand detection of the image frames every preset time interval to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand bounding box containing each hand.

[0136] The second hand position information determination module 803 is configured to determine, for each trigger image frame, hand position information of each non-trigger image frame according to hand position information of the trigger image frame, wherein the non-trigger image frame is a video frame after the trigger image frame and before a next trigger image frame.

[0137] The hand position information matching module 804 is configured to match hand position information of each hand based on hand position information of each two continuous image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is matching success or matching failure.

[0138] The dynamic gesture tracking module 805 is configured to determine a dynamic gesture in the vehicle cabin video data stream based on the matching result of the hand position information of each hand in each two continuous image frames.

[0139] In an optional embodiment, the device further comprises a position information updating module, which is configured to:

[0140] After determining, for each trigger image frame, hand position information of each non-trigger image frame according to hand position information of the trigger image frame, determine, according to the hand position information of each non-trigger image frame, hand key points of each hand in each non-trigger image frame and confidence of each hand key point.

[0141] For each non-trigger image frame, determine whether an average value of the confidence of each hand key point in the non-trigger image frame exceeds a preset threshold value.

[0142] If the average value of the confidence of each hand key point in the non-trigger image frame does not exceed the preset threshold value, trigger hand detection of the non-trigger image frame to obtain new hand position information of the non-trigger image frame.

[0143] Update the hand position information of the non-trigger image frame determined according to the hand position information of the trigger image frame by using the new hand position information of the non-trigger image frame.

[0144] In an optional embodiment, triggering, for each preset time length, hand detection of an image frame to obtain hand position information of each trigger image frame comprises:

[0145] Down-sampling each trigger image;

[0146] Input each down-sampled trigger image into a trained hand detection model to obtain hand position information of each trigger image frame.

[0147] In an optional embodiment, determining, for each trigger image frame, hand position information of each non-trigger image frame according to hand position information of the trigger image frame comprises:

[0148] The coordinate information of the hand bounding box of each hand in the trigger image frame is expanded by a preset multiple to obtain the coordinate information of the hand bounding box of each non-trigger image frame, so as to determine the hand position information of each non-trigger image frame.

[0149] In an optional embodiment, the matching of the hand position information of each hand based on the hand position information of each two continuous image frames obtains a matching result of the hand position information of each hand, and the matching result includes:

[0150] The hand key point position information of each hand in each image frame is determined based on the hand position information of each image frame.

[0151] For each two continuous image frames, a weighted bipartite graph is constructed according to the hand key point position information of each hand and the hand feature information of each hand in the two continuous image frames.

[0152] According to the weighted bipartite graph, the hand position information of each hand in the two continuous images is matched by using the KM algorithm to obtain a matching result of the hand position information of each hand.

[0153] In an optional embodiment, the determination of the hand key point position of each hand in each image frame based on the hand position information of each image frame includes:

[0154] The image region indicated by the hand position information of each image frame is cut out to obtain a hand image.

[0155] The hand image is input into a trained hand feature point regression model to obtain the hand key point position of each hand.

[0156] In an optional embodiment, the determination of the dynamic hand gesture in the vehicle cabin video data stream based on the matching result of the hand position information of each hand in each two continuous image frames includes:

[0157] For each hand, when the matching result of the hand position information of the hand in each two continuous image frames is a matching failure, the hand in the latter image frame of each two continuous image frames is taken as the start of a new hand action sequence to construct a hand action sequence.

[0158] When the matching result of the hand position information of the hand in each two continuous image frames is a matching success, the hand in each two continuous image frames is taken as a continuous part of the same hand action sequence to construct a hand action sequence.

[0159] The dynamic hand gesture in the vehicle cabin video data stream is generated according to the constructed hand action sequence.

[0160] Embodiment three

[0161] Based on the same disclosure concept, referring to FIG. 9, FIG. 9 shows a structural schematic diagram of a computer device according to the third embodiment of the present disclosure. As shown in FIG. 9, the computer device 900 according to the third embodiment of the present disclosure comprises:

[0162] The processor 901, the memory 902 and the bus 903, the memory 902 stores machine readable instructions executable by the processor 901, when the computer device 900 runs, the processor 901 and the memory 902 communicate through the bus 903, the machine readable instructions are executed by the processor 901 to perform the steps of the gesture tracking method shown in the first embodiment.

[0163] Embodiment four

[0164] Based on the same disclosure concept, the present disclosure further provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to perform the steps of the gesture tracking method described in any one of the above embodiments.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0166] The computer program product for gesture tracking provided by the embodiments of the present disclosure includes a computer readable storage medium storing program codes, the program codes include instructions for executing the method described in the foregoing method embodiments, and the specific implementation can be referred to the method embodiments, which will not be described here.

[0167] The gesture tracking device provided by the embodiments of the present disclosure can be specific hardware on the device or software or firmware installed on the device. The device provided by the embodiments of the present disclosure has the same implementation principle and technical effects as the foregoing method embodiments. For brief description, the part not mentioned in the device embodiment can refer to the corresponding content in the foregoing method embodiments. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the foregoing described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0168] In the embodiments provided by the present disclosure, it should be understood that the disclosed apparatus and method can be implemented in other manners. The embodiments described above are merely exemplary, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, and electrical, mechanical or other forms.

[0169] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0170] In addition, each functional unit in the embodiments provided by the present disclosure can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0171] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present disclosure, essentially or part of the technical solutions that make contributions to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various media that can store program codes.

[0172] It should be noted that: similar reference numerals and letters in the following drawings represent similar items, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third" and the like are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0173] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure, rather than limit the same. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any modifications or easy-to-think changes to the technical solutions recorded in the foregoing embodiments, or equivalent replacements of some technical features, within the technical scope disclosed by the present disclosure, do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure. All should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims. Industrial applicability

[0174] The present disclosure can reduce the number of image frames that need to be detected, reduce the number of hand detection, and only detect the hand of the trigger image frame, thereby reducing the required computing power when tracking gestures.

[0175] Then, for each trigger image frame, the hand position information of each non-trigger image frame is determined according to the hand position information of the trigger image frame, which can determine the hand position information in the non-trigger image frame without the need for hand detection of other non-trigger image frames that do not need to be detected, thereby providing complete hand position information of each image frame for gesture detection.

[0176] Finally, the hand position information of each hand is matched based on the hand position information of each two consecutive image frames to obtain the matching result of the hand position information of each hand, and the dynamic gesture in the vehicle cabin video data stream is determined based on the matching result of the hand position information of each hand in each two consecutive image frames, which can track and identify the dynamic gesture in the vehicle cabin video according to the matching result of the hand position information in each image frame, thereby realizing low-power gesture tracking in the vehicle cabin.

[0177] The above scheme limits the number of times of triggering hand detection, and only calls the model to perform hand detection on the image frame triggering hand detection, without using the model to perform hand detection on the image frame not triggering hand detection, thereby greatly reducing the calculation power consumption during gesture tracking. Meanwhile, the hand detection result of the image frame not triggering hand detection is determined based on the hand detection result of the image frame triggering hand detection, which can provide complete and accurate hand data for subsequent hand matching, effectively reduce the missed detection of hands, and improve the accuracy of hand detection. In the gesture tracking process, the non-triggering image frame with inaccurate hand detection result is also re-detected, which can reduce the probability of hand tracking failure caused by too fast hand movement, and improve the accuracy of the hand detection result. After triggering detection, the bipartite graph is established through a specific weight, and the optimal matching is solved through the KM algorithm, which can accurately maintain the timing information of the hand, thereby reducing the calculation power consumed during gesture tracking, reducing the requirement for the calculation ability of the device during gesture tracking, and reducing the limitation of gesture tracking while ensuring accurate gesture tracking through various devices.

[0178] In addition, it can be understood that the gesture tracking method, device, computer device and readable storage medium provided by the embodiments of the present disclosure are reproducible and can be used in various industrial applications. For example, the gesture tracking method, device, computer device and readable storage medium provided by the embodiments of the present disclosure can be used in the field of computer technology.

Claims

1. A gesture tracking method, characterized by, The method comprises: acquiring a vehicle cabin video data stream, and performing image capture on the vehicle cabin video data stream to obtain a plurality of image frames; triggering hand detection on the image frames at a preset time interval to obtain hand position information of each triggered image frame, wherein the hand position information comprises coordinate information of a hand bounding box of each hand; for each triggered image frame, determining hand position information of each non-triggered image frame according to the hand position information of the triggered image frame, wherein the non-triggered image frame is a video frame after the triggered image frame and before a next triggered image frame; matching the hand position information of each hand based on the hand position information of each two continuous image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is matching success or matching failure; determining a dynamic gesture in the vehicle cabin video data stream based on the matching result of the hand position information of each hand in each two continuous image frames.

2. The method of claim 1, wherein, After determining the hand position information of each non-triggered image frame according to the hand position information of the triggered image frame, the method further comprises: determining hand key points of each hand and confidence of each hand key point in each non-triggered image frame according to the hand position information of each non-triggered image frame; for each non-triggered image frame, judging whether an average value of the confidence of each hand key point in the non-triggered image frame exceeds a preset threshold value; if the average value of the confidence of each hand key point in the non-triggered image frame does not exceed the preset threshold value, triggering hand detection on the non-triggered image frame to obtain new hand position information of the non-triggered image frame; updating the hand position information of the non-triggered image frame determined according to the hand position information of the triggered image frame by using the new hand position information of the non-triggered image frame.

3. The method of claim 1, wherein, The method of triggering hand detection on the image frames at a preset time interval to obtain the hand position information of each triggered image frame comprises: performing downsampling processing on each triggered image; inputting each triggered image after the downsampling processing into a trained hand detection model to obtain the hand position information of each triggered image frame.

4. The method of claim 1, wherein, The method of determining the hand position information of each non-triggered image frame according to the hand position information of the triggered image frame comprises: extending the coordinate information of the hand bounding box of each hand in the triggered image frame by a preset multiple to obtain coordinate information of a hand bounding box of each non-triggered image frame, so as to determine the hand position information of each non-triggered image frame.

5. The method of claim 1, wherein, The method of matching the hand position information of each hand based on the hand position information of each two continuous image frames to obtain a matching result of the hand position information of each hand comprises: determining hand key point position information of each hand in each image frame based on the hand position information of each image frame; for each two continuous image frames, constructing a weighted bipartite graph according to the hand key point position information of each hand and the hand property information of each hand in the two continuous image frames; matching the hand position information of each hand in the two continuous images according to the weighted bipartite graph by using a KM algorithm to obtain the matching result of the hand position information of each hand.

6. The method of claim 5, wherein, The hand position information of each image frame determines the hand key point position of each hand in each image frame, including: The image region indicated by the hand position information of each image frame is cut out to obtain each hand image; The hand key point position of each hand is obtained by inputting each hand image into the trained hand feature point regression model.

7. The method of claim 1, wherein, The matching result of the hand position information of each hand in each two continuous image frames is used to determine the dynamic hand gesture in the vehicle cabin video data stream, including: For each hand, when the matching result of the hand position information of the hand in each two continuous image frames is a matching failure, the hand in the latter image frame in each two continuous image frames is taken as the start of a new hand action sequence to construct a hand action sequence; When the matching result of the hand position information of the hand in each two continuous image frames is a matching success, the hand in each two continuous image frames is taken as a continuous part of the same hand action sequence to construct a hand action sequence; The dynamic hand gesture in the vehicle cabin video data stream is generated according to the constructed hand action sequence.

8. A dynamic gesture tracking apparatus, characterized by, The device includes: An image frame acquisition module is configured to acquire a vehicle cabin video data stream, and capture images from the vehicle cabin video data stream to obtain a plurality of image frames; A first hand position information determination module is configured to trigger hand detection of image frames every preset time length to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand bounding box of each hand; A second hand position information determination module is configured to determine hand position information of each non-triggered image frame according to the hand position information of the triggered image frame for each triggered image frame, wherein the non-triggered image frame is a video frame after the triggered image frame and before the next triggered image frame; A hand position information matching module is configured to match the hand position information of each hand based on the hand position information of each two continuous image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure; A dynamic hand gesture tracking module is configured to determine the dynamic hand gesture in the vehicle cabin video data stream based on the matching result of the hand position information of each hand in each two continuous image frames.

9. A computer device, comprising: It includes: A processor, a memory, and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the gesture tracking method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which is executed by the processor to execute the steps of the gesture tracking method in any one of claims 1-7.

Images