A hand gesture tracking method, device, computer equipment and readable storage medium
By triggering hand detection at every preset time in gesture tracking technology and updating hand position based on matching results, combining downsampling and feature point regression models, the problem of high requirements for computing resources in the existing technology is solved, and the calculation is realized Efficient gesture tracking on weaker devices.
Patent Information
- Application Number
- CN202411118583.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-15
AI Technical Summary
The existing gesture tracking technology has high requirements for computing resources and is difficult to implement gesture tracking on devices with weak computing performance, limiting its application scope.
By triggering hand detection every preset time, the hand position information is obtained, and dynamic gestures are determined based on the matching results of continuous frames, unnecessary detection times are reduced, and downsampling and feature point regression models are used to improve accuracy.
It reduces the computing power consumption required for gesture tracking, improves the accuracy of hand detection and the computing power utilization of the device, and ensures the gesture tracking function on multiple devices.
Smart Images

Figure CN118644525B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a gesture tracking method, device, computer equipment and readable storage medium. Background Art
[0002] With the rapid development of the automotive industry, smart cockpit technology has become a key factor in enhancing the competitiveness of modern cars. The smart cockpit is not just a driving space, it is also an advanced human-computer interaction platform that integrates various sensors and interactive interfaces. On this platform, multi-hand tracking and matching technologies are particularly important because they are directly related to the user's operational convenience and safety. Multi-hand tracking technology allows the system to monitor and identify the hands of the driver and passengers in real time through the camera. The maintained hand timing action information is converted into corresponding commands such as adjusting the volume, switching playing tracks, answering calls, etc. This interactive method without physical contact greatly enriches the experience of drivers and passengers, while also reducing distractions during driving and enhancing driving safety.
[0003] In the prior art, when tracking gestures, the gesture detection processing schemes adopted all use a deep learning model to detect and process each frame of the video stream to output the hand gesture frame information, and then perform gesture tracking based on the gesture frame information output by the model. However, it was found in the research that since it is necessary to call the deep learning model to detect and process each frame of the image, it takes a lot of computing power, so the existing gesture detection processing scheme has high requirements on the computing power of the device, has great limitations, and is difficult to apply to devices with weaker computing performance. For some cabin systems and equipment with weaker computing performance, it is impossible to provide sufficient computing resources for multiple model runs, and gesture tracking cannot be achieved through the existing gesture detection processing. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a gesture tracking method, apparatus, computer equipment and readable storage medium to reduce the computing power required for gesture tracking, reduce the requirements for device computing power when performing gesture tracking, reduce the limitations of gesture tracking, and ensure that gesture tracking can be performed normally through various devices.
[0005] In a first aspect, an embodiment of the present application provides a gesture tracking method, the method comprising:
[0006] Acquire a vehicle cabin video data stream, and perform image capture on the vehicle cabin video data stream to obtain a plurality of image frames;
[0007] Triggering hand detection on the image frame at a preset time interval to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand circumscribed frame containing each hand;
[0008] For each trigger image frame, the hand position information of each non-trigger image frame is determined according to the hand position information of the trigger image frame, wherein the non-trigger image frame is a video frame after the trigger image frame and before the next trigger image frame;
[0009] Matching the hand position information of each hand based on the hand position information of every two consecutive image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure;
[0010] The dynamic gesture in the cabin video data stream is determined based on the matching result of the hand position information of each hand in every two consecutive image frames.
[0011] Optionally, after determining the hand position information of each non-triggering image frame according to the hand position information of the triggering image frame, the method further includes:
[0012] Determine the hand key points of each hand in each non-trigger image frame and the confidence of each hand key point according to the hand position information of each non-trigger image frame;
[0013] For each non-trigger image frame, determining whether the average confidence level of each hand key point in the non-trigger image frame exceeds a preset threshold;
[0014] If the average value of the confidence of the hand key points of each hand in the non-trigger image frame does not exceed the preset threshold, triggering hand detection of the non-trigger image frame to obtain new hand position information of the non-trigger image frame;
[0015] The hand position information of the non-trigger image frame determined based on the hand position information of the trigger image frame is updated using the new hand position information of the non-trigger image frame.
[0016] Optionally, triggering hand detection on an image frame at a preset time interval to obtain hand position information of each triggered image frame includes:
[0017] Perform down-sampling processing on each triggered image;
[0018] Each trigger image after downsampling is input into the trained hand detection model to obtain the hand position information of each trigger image frame.
[0019] Optionally, determining the hand position information of each non-triggering image frame according to the hand position information of the triggering image frame includes:
[0020] The coordinate information of the hand circumference frame of each hand in the trigger image frame is expanded by a preset multiple to obtain the coordinate information of the hand circumference frame of each non-trigger image frame, so as to determine the hand position information of each non-trigger image frame.
[0021] Optionally, the matching of the hand position information of each hand based on the hand position information of every two consecutive image frames to obtain the matching result of the hand position information of each hand includes:
[0022] Determine the hand key point position information of each hand in each image frame based on the hand position information of each image frame;
[0023] For every two consecutive image frames, construct a weighted bipartite graph according to the hand key point position information of each hand and the chirality information of each hand in the two consecutive image frames;
[0024] According to the weighted bipartite graph, the hand position information of each hand in the two consecutive images is matched using the KM algorithm to obtain a matching result of the hand position information of each hand.
[0025] Optionally, determining the hand key point position of each hand in each image frame based on the hand position information of each image frame includes:
[0026] Cut out the image area indicated by the hand position information of each image frame to obtain each hand image;
[0027] Each hand image is input into the trained hand feature point regression model to obtain the hand key point position of each hand.
[0028] Optionally, the determining the dynamic gesture in the cabin video data stream based on the matching result of the hand position information of each hand in every two consecutive image frames includes:
[0029] For each hand, when the matching result of the hand position information of the hand in every two consecutive image frames is a matching failure, the hand in the latter image frame of every two consecutive image frames is used as the beginning of a new hand action sequence to construct a hand action sequence;
[0030] When the matching result of the hand position information of the hand in every two consecutive image frames is a successful match, the hand in every two consecutive image frames is taken as a continuous part of the same hand action sequence to construct a hand action sequence;
[0031] Dynamic gestures in the cabin video data stream are generated according to the constructed hand movement sequence.
[0032] In a second aspect, an embodiment of the present application provides a dynamic gesture tracking device, the device comprising:
[0033] An image frame acquisition module, used to acquire a vehicle cabin video data stream, and to capture images of the vehicle cabin video data stream to obtain a plurality of image frames;
[0034] A first hand position information determination module is used to trigger hand detection of image frames at preset time intervals to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand circumscribed frame containing each hand;
[0035] A second hand position information determination module is used to determine the hand position information of each non-trigger image frame according to the hand position information of the trigger image frame, wherein the non-trigger image frame is a video frame after the trigger image frame and before the next trigger image frame;
[0036] A hand position information matching module, used for matching the hand position information of each hand based on the hand position information of every two consecutive image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure;
[0037] The dynamic gesture tracking module is used to determine the dynamic gesture in the cabin video data stream based on the matching result of the hand position information of each hand in every two consecutive image frames.
[0038] Optionally, the device further includes a location information updating module, and the location information updating module is used to:
[0039] After determining the hand position information of each non-trigger image frame according to the hand position information of the trigger image frame for each trigger image frame, determine the hand key points of each hand in each non-trigger image frame and the confidence of each hand key point according to the hand position information of each non-trigger image frame;
[0040] For each non-trigger image frame, determining whether the average confidence level of each hand key point in the non-trigger image frame exceeds a preset threshold;
[0041] If the average value of the confidence of the hand key points of each hand in the non-trigger image frame does not exceed the preset threshold, triggering hand detection of the non-trigger image frame to obtain new hand position information of the non-trigger image frame;
[0042] The hand position information of the non-trigger image frame determined based on the hand position information of the trigger image frame is updated using the new hand position information of the non-trigger image frame.
[0043] Optionally, triggering hand detection on an image frame at a preset time interval to obtain hand position information of each triggered image frame includes:
[0044] Perform down-sampling processing on each triggered image;
[0045] Each trigger image after downsampling is input into the trained hand detection model to obtain the hand position information of each trigger image frame.
[0046] Optionally, determining the hand position information of each non-triggering image frame according to the hand position information of the triggering image frame includes:
[0047] The coordinate information of the hand circumference frame of each hand in the trigger image frame is expanded by a preset multiple to obtain the coordinate information of the hand circumference frame of each non-trigger image frame, so as to determine the hand position information of each non-trigger image frame.
[0048] Optionally, the matching of the hand position information of each hand based on the hand position information of every two consecutive image frames to obtain the matching result of the hand position information of each hand includes:
[0049] Determine the hand key point position information of each hand in each image frame based on the hand position information of each image frame;
[0050] For every two consecutive image frames, construct a weighted bipartite graph according to the hand key point position information of each hand and the chirality information of each hand in the two consecutive image frames;
[0051] According to the weighted bipartite graph, the hand position information of each hand in the two consecutive images is matched using the KM algorithm to obtain a matching result of the hand position information of each hand.
[0052] Optionally, determining the hand key point position of each hand in each image frame based on the hand position information of each image frame includes:
[0053] Cut out the image area indicated by the hand position information of each image frame to obtain each hand image;
[0054] Each hand image is input into the trained hand feature point regression model to obtain the hand key point position of each hand.
[0055] Optionally, the determining the dynamic gesture in the cabin video data stream based on the matching result of the hand position information of each hand in every two consecutive image frames includes:
[0056] For each hand, when the matching result of the hand position information of the hand in every two consecutive image frames is a matching failure, the hand in the latter image frame of every two consecutive image frames is used as the beginning of a new hand action sequence to construct a hand action sequence;
[0057] When the matching result of the hand position information of the hand in every two consecutive image frames is a successful match, the hand in every two consecutive image frames is taken as a continuous part of the same hand action sequence to construct a hand action sequence;
[0058] Dynamic gestures in the cabin video data stream are generated according to the constructed hand movement sequence.
[0059] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the gesture tracking method described in any optional implementation manner of the first aspect are performed.
[0060] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the gesture tracking method described in any optional implementation manner in the above-mentioned first aspect are executed.
[0061] The technical solution provided by this application includes but is not limited to the following beneficial effects:
[0062] The present application obtains a vehicle cabin video data stream, performs image capture on the vehicle cabin video data stream to obtain a number of image frames, and then triggers hand detection on the image frames at preset time intervals to obtain hand position information for each triggered image frame. This can reduce the number of image frames that require hand detection and the number of hand detections. Hand detection is performed only on the triggered image frames, thereby reducing the computing power required for gesture tracking.
[0063] Then, for each triggered image frame, the hand position information of each non-triggered image frame is determined based on the hand position information of the trigger image frame. This allows the hand position information in the non-triggered image frames to be determined without the need to perform hand detection on other non-triggered image frames that have not performed hand detection, thereby providing complete hand position information for each image frame for gesture detection.
[0064] Finally, the hand position information of each hand is matched based on the hand position information of every two consecutive image frames to obtain the matching result of the hand position information of each hand, and the dynamic gestures in the cabin video data stream are determined based on the matching result of the hand position information of each hand in every two consecutive image frames. It is possible to track, identify and detect the dynamic gestures in the cabin video according to the matching results of the hand position information in each image frame, thereby realizing low-computing-power in-cabin gesture tracking.
[0065] By adopting the above scheme, the number of triggers of hand detection is limited, and the model is only called to perform hand detection on the image frames that trigger hand detection, and the model is not used to perform hand detection on the image frames that do not trigger hand detection, thereby greatly reducing the computing power consumption during gesture tracking. At the same time, the hand detection results of the image frames that do not trigger hand detection are determined based on the hand detection results of the image frames that trigger hand detection, which can provide complete and accurate hand data for subsequent hand matching, can effectively reduce the missed detection of hands, and improve the accuracy of hand detection. In the gesture tracking process, hand detection will be performed again for some non-triggered image frames with inaccurate hand detection results, which can reduce the probability of the hand moving too fast and causing the hand to fail to track, and improve the accuracy of the hand detection results. After the trigger detection, a bipartite graph is established through specific weights, and the optimal match is solved through the KM algorithm to accurately maintain the timing information of the hand, thereby reducing the computing power consumed during gesture tracking, reducing the requirements for the computing power of the device during gesture tracking, reducing the limitations of gesture tracking, and ensuring that accurate gesture tracking can be performed through various devices.
[0066] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0068] Figure 1 A flowchart of a gesture tracking method provided by Embodiment 1 of the present invention is shown;
[0069] Figure 2 A flow chart of a location information updating method provided by Embodiment 1 of the present invention is shown;
[0070] Figure 3 A flow chart of a method for determining location information provided by Embodiment 1 of the present invention is shown;
[0071] Figure 4 A flow chart of a hand position information matching method provided in Embodiment 1 of the present invention is shown;
[0072] Figure 5 A flow chart of a method for locating key points of a hand provided in the first embodiment of the present invention is shown;
[0073] Figure 6A flow chart of a dynamic gesture determination method provided by Embodiment 1 of the present invention is shown;
[0074] Figure 7 A flowchart of a gesture tracking method in a smart cockpit provided by Embodiment 1 of the present invention is shown;
[0075] Figure 8 A schematic diagram of the structure of a dynamic gesture tracking device provided by Embodiment 2 of the present invention is shown;
[0076] Fig. 9 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION
[0077] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.
[0078] Embodiment 1
[0079] To facilitate understanding of this application, Figure 1 The flowchart of a gesture tracking method provided in the first embodiment of the present invention is shown to describe the contents of the first embodiment of the present application in detail.
[0080] See also Figure 1 As shown, Figure 1 A flowchart of a gesture tracking method provided in the first embodiment of the present invention is shown, wherein the method comprises steps S101 to S104:
[0081] S101: Acquire a vehicle cabin video data stream, and perform image capture on the vehicle cabin video data stream to obtain a plurality of image frames.
[0082] Specifically, when a cabin occupant enters the cabin, the cabin video data stream is collected by a camera, and hand detection is performed on each image frame.
[0083] S102: triggering hand detection on image frames at preset time intervals to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand circumscribed frame containing each hand.
[0084] Specifically, hand detection of image frames is triggered once every 500 ms, and the image frames triggered by hand detection are recorded as triggering image frames. For each triggering image frame, hand detection is performed on the triggering image frame using the Nanodet-Plus hand detection model to obtain the hand position information of the triggering image frame, that is, the coordinate information of the hand external bounding box of each hand contained in the triggering image frame, that is, the horizontal and vertical coordinates of the upper left corner and the lower right corner of the hand external bounding box.
[0085] S103: For each trigger image frame, determine the hand position information of each non-trigger image frame according to the hand position information of the trigger image frame, wherein the non-trigger image frame is a video frame after the trigger image frame and before the next trigger image frame.
[0086] Specifically, since the hand detection is performed at preset time intervals, there are several non-trigger image frames without hand detection between each two hand detections, that is, between each two adjacent trigger image frames. For these non-trigger image frames without hand detection, the hand position information of the non-trigger image is determined based on the hand position information of the most recent trigger image frame in the history of the non-trigger image frame.
[0087] S104: Matching the hand position information of each hand based on the hand position information of every two consecutive image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure.
[0088] Specifically, for each image frame, the hand position information of the image frame is matched with the hand position information of the previous image frame through weighted bipartite graph optimal matching to obtain the optimal matching result.
[0089] S105: Determine the dynamic gesture in the cabin video data stream based on the matching result of the hand position information of each hand in every two consecutive image frames.
[0090] Specifically, for each image frame, if the hand position information of the image frame fails to match the hand position information of the previous image frame of the image frame (i.e., the match fails), then the hand that failed to match in the image frame will be assigned a new hand identifier, indicating that this is the beginning of a new hand movement sequence. If the hand position information of the image frame matches the hand position information of the previous image frame of the image frame, then the two successfully matched hands are regarded as continuous parts of the same hand movement sequence, and the hand of the image frame will continue to maintain the hand identifier of the previous image frame of the image frame. The beginning of a new hand movement sequence is regarded as the beginning of a dynamic gesture, and the continuous parts of each hand movement sequence are regarded as components after the start of a dynamic gesture, so as to construct a dynamic gesture.
[0091] In an alternative embodiment, see Figure 2 As shown, Figure 2 A flow chart of a position information updating method provided by the first embodiment of the present invention is shown, wherein, for each triggering image frame, after determining the hand position information of each non-triggering image frame according to the hand position information of the triggering image frame, the method further includes steps S201 to S204:
[0092] S201: Determine the hand key points of each hand in each non-trigger image frame and the confidence level of each hand key point according to the hand position information of each non-trigger image frame.
[0093] Specifically, according to the hand position information of each non-trigger image frame, a hand image containing each hand is extracted, and then each hand image is input into the hand 21-point feature point regression model to obtain the specific position of the 21 hand skeleton points in each non-trigger image and the confidence of each point. The hand 21-point feature point regression model uses a lightweight QaRepVgg feature extraction network and uses two loss functions for training. RLEloss is used for the task of locating the key points of the hand and outputs the coordinates and confidence of each key point of the hand, while CrossEntropyLoss is used to determine the handedness of the hand and output the left-right nature of the hand, that is, the left hand or the right hand.
[0094] S202: For each non-trigger image frame, determine whether the average confidence value of each hand key point in the non-trigger image frame exceeds a preset threshold.
[0095] Specifically, for each non-trigger image frame, it is calculated whether the average value of the confidence of each hand key point in the non-trigger image frame exceeds a preset threshold, such as 0.2.
[0096] S203: If the average confidence value of the hand key points of each hand in the non-trigger image frame does not exceed the preset threshold, trigger hand detection on the non-trigger image frame to obtain new hand position information of the non-trigger image frame.
[0097] Specifically, if the average confidence value of the hand key points of each hand in the non-trigger image frame does not exceed the preset threshold, it is considered that the key point detection of the non-trigger image frame is inaccurate, then the hand detection of the non-trigger image frame is triggered, and the Nanodet-Plus hand detection model is used to perform hand detection on the non-trigger image frame to obtain accurate new hand position information.
[0098] S204: Using the new hand position information of the non-trigger image frame, the hand position information of the non-trigger image frame is updated according to the hand position information of the trigger image frame.
[0099] Specifically, the hand position information of the non-trigger image frame obtained in step S103 is updated using the new hand position information of the non-trigger image frame, so that step S104 performs hand matching according to the new hand position information of the non-trigger image frame.
[0100] If the average confidence value of the hand key points of each hand in the non-trigger image frame exceeds the preset threshold, it is considered that the key point detection of the non-trigger image frame is accurate and available, and there is no need to trigger hand detection on the non-trigger image frame. Hand matching is performed directly according to the hand position information of the non-trigger image frame obtained in step S103.
[0101] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flow chart of a method for determining position information provided by the first embodiment of the present invention is shown, wherein the hand detection of the image frame is triggered at a preset time interval to obtain the hand position information of each triggered image frame, including steps S301-S302:
[0102] S301: Perform down-sampling processing on each trigger image.
[0103] Specifically, down-sampling processing is performed on each trigger image to reduce the image size.
[0104] S302: Input each trigger image after downsampling processing into the trained hand detection model to obtain the hand position information of each trigger image frame.
[0105] Specifically, the hand detection model is a Nanodet-Plus hand detection model.
[0106] In an optional implementation, determining the hand position information of each non-triggering image frame according to the hand position information of the triggering image frame includes:
[0107] The coordinate information of the hand circumference frame of each hand in the trigger image frame is expanded by a preset multiple to obtain the coordinate information of the hand circumference frame of each non-trigger image frame, so as to determine the hand position information of each non-trigger image frame.
[0108] Specifically, for each non-trigger image frame, the hand external bounding box of the most recent trigger image frame in history of the non-trigger image frame is expanded to obtain the hand external bounding box of the non-trigger image frame as the hand position information of the non-trigger image frame. The most recent trigger image frame in history is the trigger image frame that is before the non-trigger image and closest to the non-trigger image.
[0109] In an alternative embodiment, see Figure 4 As shown, Figure 4The flowchart of a hand position information matching method provided by Embodiment 1 of the present invention is shown. Among them, the hand position information of each hand is matched based on the hand position information of every two consecutive image frames to obtain the matching result of the hand position information of each hand, including steps S401 to S403:
[0110] S401: Determine the hand key point position information of each hand in each image frame based on the hand position information of each image frame.
[0111] Specifically, the hand images containing each hand are cropped according to the hand bounding box information of each image frame, and then the hand images are input into the hand 21-point feature point regression model to obtain the position information of 21 hand key points in each image frame and the confidence of each hand key point.
[0112] S402: For every two consecutive image frames, construct a weighted bipartite graph according to the hand key point position information and the chirality information of each hand in the two consecutive image frames.
[0113] Specifically, the method for establishing the weighted bipartite graph is as follows:
[0114] For every two consecutive image frames, denote the set of hands detected in the latter image frame of the two consecutive image frames as A; the set of hands in the former image frame of the two consecutive image frames, that is, the set of hands maintained within a previously preset length (optionally 3s) window, as B. A and B are respectively used as the left and right parts of the bipartite graph. For each element a in A and each element b in B, calculate the distance between each a and each b according to the hand key point position information of each hand. If the distance between a and b is less than the preset threshold D, then connect a and b, and make the edge weight W(a, b) between a and b be: W(a, b) = 1 / (x * dist(a, b) + K), where if both a and b are left hands or right hands, then x is the first preset parameter X0, otherwise x is the second preset parameter X1 (where 0 < X0 < X1), dist(a, b) represents the Euclidean distance between the wrist key point coordinates of a and b, and k is a preset fixed parameter (a positive number).
[0115] S403: According to the weighted bipartite graph, use the KM algorithm to match the hand position information of each hand in the two consecutive images to obtain the matching result of the hand position information of each hand.
[0116] Specifically, based on the weighted bipartite graph constructed by the above method, the hand position information of each hand in the two consecutive images is matched by the KM algorithm, and the matching results include successful matching and failed matching. When the hand position information of a certain hand in the two consecutive images matches each other, it means that the matching result of the hand is a successful matching; when the hand position information of a certain hand in the two consecutive images does not match each other, it means that the matching result of the hand is a failed matching.
[0117] In an alternative embodiment, see Figure 5 As shown, Figure 5 A flow chart of a hand key point positioning method provided by the first embodiment of the present invention is shown, wherein the hand key point position of each hand in each image frame is determined based on the hand position information of each image frame, including steps S501-S502:
[0118] S501: Extracting the image area indicated by the hand position information of each image frame to obtain each hand image.
[0119] S502: Input each hand image into the trained hand feature point regression model to obtain the hand key point position of each hand.
[0120] Specifically, the hand feature point regression model can also output the confidence of each hand key point and the handedness of each hand.
[0121] In an alternative embodiment, see Figure 6 As shown, Figure 6 A flow chart of a method for determining a dynamic gesture provided by the first embodiment of the present invention is shown, wherein the dynamic gesture in the cabin video data stream is determined based on the matching result of the hand position information of each hand in every two consecutive image frames, including steps S601 to S603:
[0122] S601: For each hand, when the matching result of the hand position information of the hand in every two consecutive image frames is a matching failure, the hand in the latter image frame between every two consecutive image frames is used as the beginning of a new hand movement sequence to construct a hand movement sequence.
[0123] Specifically, a new hand identifier is assigned to the hand.
[0124] S602: When the matching result of the hand position information of the hand in every two consecutive image frames is a successful match, the hand in every two consecutive image frames is taken as a continuous part of the same hand action sequence to construct a hand action sequence.
[0125] Specifically, the hand is assigned the same hand identifier as that of the previous image frame, that is, the hand identifier of the previous image frame is continued.
[0126] S603: Generate dynamic gestures in the cabin video data stream according to the constructed hand movement sequence.
[0127] Specifically, hands with the same hand identifiers are regarded as components of the same hand movement sequence. The hand identifiers in different hand movement sequences are different, and the hand identifiers in the same hand movement sequence are the same. For each hand movement sequence, the hands in the hand movement sequence are tracked and identified to obtain the dynamic gestures of the hands.
[0128] When the gesture tracking method provided in this application is applied to the intelligent cockpit to track and identify the gestures of the cabin occupants, see Figure 7 As shown, Figure 7 The flowchart of a gesture tracking method in a smart cockpit provided by the first embodiment of the present invention is shown, wherein the following steps are specifically included: real-time acquisition of the current frame video stream data, detection of whether no hand is detected in the previous frame; if so, triggering hand detection; if not, determining whether the time from the last triggering of hand detection is greater than 500ms, if so, triggering hand detection, if not, expanding the external frame of the 21 hand key points solved by the previous frame detection to obtain the external frame of the current frame, and then calculating the key points of the hand and the confidence of each key point in the current frame based on the external frame of the current frame; determining whether the average value of the confidence of each key point is less than a preset threshold, if so, triggering hand detection, if not, ending hand tracking. When triggering hand detection, determine the key points of the hand, the confidence of each hand key point, and the left and right hand classification information of each hand, and then establish an adjacency matrix through the external frame of the current frame and the information of the hand that appeared most recently within the history of 3s to form a weighted bipartite graph, and then solve the hand matched by the external frame of the current frame through the KM algorithm, and complete and end hand tracking.
[0129] Embodiment 2
[0130] See also Figure 8 As shown, Figure 8 A schematic diagram of the structure of a dynamic gesture tracking device provided by Embodiment 2 of the present invention is shown, wherein the device comprises:
[0131] An image frame acquisition module 801 is used to acquire a vehicle cabin video data stream, and to capture images of the vehicle cabin video data stream to obtain a plurality of image frames;
[0132] A first hand position information determination module 802 is used to trigger hand detection of image frames at preset time intervals to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand circumscribed frame containing each hand;
[0133] The second hand position information determination module 803 is used to determine the hand position information of each non-trigger image frame according to the hand position information of the trigger image frame, wherein the non-trigger image frame is a video frame after the trigger image frame and before the next trigger image frame;
[0134] The hand position information matching module 804 is used to match the hand position information of each hand based on the hand position information of every two consecutive image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure;
[0135] The dynamic gesture tracking module 805 is used to determine the dynamic gesture in the cabin video data stream based on the matching result of the hand position information of each hand in every two consecutive image frames.
[0136] In an optional implementation, the device further includes a location information updating module, wherein the location information updating module is configured to:
[0137] After determining the hand position information of each non-trigger image frame according to the hand position information of the trigger image frame for each trigger image frame, determine the hand key points of each hand in each non-trigger image frame and the confidence of each hand key point according to the hand position information of each non-trigger image frame;
[0138] For each non-trigger image frame, determining whether the average confidence level of each hand key point in the non-trigger image frame exceeds a preset threshold;
[0139] If the average value of the confidence of the hand key points of each hand in the non-trigger image frame does not exceed the preset threshold, triggering hand detection of the non-trigger image frame to obtain new hand position information of the non-trigger image frame;
[0140] The hand position information of the non-trigger image frame determined based on the hand position information of the trigger image frame is updated using the new hand position information of the non-trigger image frame.
[0141] In an optional implementation, triggering hand detection on an image frame at a preset time interval to obtain hand position information of each triggered image frame includes:
[0142] Perform down-sampling processing on each triggered image;
[0143] Each trigger image after downsampling is input into the trained hand detection model to obtain the hand position information of each trigger image frame.
[0144] In an optional implementation, determining the hand position information of each non-triggering image frame according to the hand position information of the triggering image frame includes:
[0145] The coordinate information of the hand circumference frame of each hand in the trigger image frame is expanded by a preset multiple to obtain the coordinate information of the hand circumference frame of each non-trigger image frame, so as to determine the hand position information of each non-trigger image frame.
[0146] In an optional implementation, the matching result of the hand position information of each hand is obtained by matching the hand position information of each hand based on the hand position information of every two consecutive image frames, including:
[0147] Determine the hand key point position information of each hand in each image frame based on the hand position information of each image frame;
[0148] For every two consecutive image frames, construct a weighted bipartite graph according to the hand key point position information of each hand and the chirality information of each hand in the two consecutive image frames;
[0149] According to the weighted bipartite graph, the hand position information of each hand in the two consecutive images is matched using the KM algorithm to obtain a matching result of the hand position information of each hand.
[0150] In an optional implementation, determining the hand key point position of each hand in each image frame based on the hand position information of each image frame includes:
[0151] Cut out the image area indicated by the hand position information of each image frame to obtain each hand image;
[0152] Each hand image is input into the trained hand feature point regression model to obtain the hand key point position of each hand.
[0153] In an optional embodiment, the determining the dynamic gesture in the cabin video data stream based on the matching result of the hand position information of each hand in every two consecutive image frames includes:
[0154] For each hand, when the matching result of the hand position information of the hand in every two consecutive image frames is a matching failure, the hand in the latter image frame of every two consecutive image frames is used as the beginning of a new hand action sequence to construct a hand action sequence;
[0155] When the matching result of the hand position information of the hand in every two consecutive image frames is a successful match, the hand in every two consecutive image frames is taken as a continuous part of the same hand action sequence to construct a hand action sequence;
[0156] Dynamic gestures in the cabin video data stream are generated according to the constructed hand movement sequence.
[0157] Embodiment 3
[0158] Based on the same application concept, see Fig. 9 As shown, Fig. 9 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Fig. 9 As shown, a computer device 900 provided in Embodiment 3 of the present application includes:
[0159] A processor 901, a memory 902 and a bus 903, wherein the memory 902 stores machine-readable instructions executable by the processor 901. When the computer device 900 is running, the processor 901 communicates with the memory 902 via the bus 903, and the machine-readable instructions are executed by the processor 901 to execute the steps of the gesture tracking method shown in the first embodiment.
[0160] Embodiment 4
[0161] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the gesture tracking method described in any one of the above embodiments are executed.
[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0163] The computer program product for performing gesture tracking provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.
[0164] The gesture tracking device provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.
[0165] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0167] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0168] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0169] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.
[0170] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A gesture tracking method, characterized in that: The method comprises: Acquiring a vehicle cabin video data stream, and performing image capture on the vehicle cabin video data stream to obtain a plurality of image frames; Triggering hand detection on image frames at preset time intervals to obtain hand position information of each triggered image frame, wherein the hand position information is coordinate information of a hand circumscribed frame containing each hand; For each trigger image frame, determining the hand position information of each non-trigger image frame based on the hand position information of the trigger image frame, wherein the non-trigger image frame is a video frame after the trigger image frame and before the next trigger image frame; Matching the hand position information of each hand based on the hand position information of every two consecutive image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure; determining a dynamic gesture in the vehicle cabin video data stream based on a matching result of hand position information of each hand in every two consecutive image frames; After determining the hand position information of each non-triggering image frame based on the hand position information of the triggering image frame, the method further includes: Determining the hand key points of each hand in each non-trigger image frame and the confidence level of each hand key point according to the hand position information of each non-trigger image frame; For each non-trigger image frame, determining whether the average confidence value of each hand key point in the non-trigger image frame exceeds a preset threshold; If the average confidence value of the hand key points of each hand in the non-trigger image frame does not exceed the preset threshold, triggering hand detection on the non-trigger image frame to obtain new hand position information of the non-trigger image frame; The hand position information of the non-trigger image frame determined based on the hand position information of the trigger image frame is updated using the new hand position information of the non-trigger image frame.
2. The method according to claim 1, characterized in that The method of triggering hand detection on an image frame at a preset time interval to obtain hand position information of each triggered image frame includes: Perform down-sampling processing on each triggered image; Each trigger image after downsampling is input into the trained hand detection model to obtain the hand position information of each trigger image frame.
3. The method according to claim 1, characterized in that The determining of the hand position information of each non-triggering image frame based on the hand position information of the triggering image frame includes: The coordinate information of the hand circumscribed frame of each hand in the trigger image frame is expanded by a preset multiple to obtain the coordinate information of the hand circumscribed frame of each non-trigger image frame, so as to determine the hand position information of each non-trigger image frame.
4. The method according to claim 1, wherein The matching result of the hand position information of each hand is obtained by matching the hand position information of each hand based on the hand position information of every two consecutive image frames, including: Determining hand key point position information of each hand in each image frame based on the hand position information of each image frame; For every two consecutive image frames, construct a weighted bipartite graph based on the hand key point position information and the chirality information of each hand in the two consecutive image frames; According to the weighted bipartite graph, the hand position information of each hand in the two consecutive images is matched using the KM algorithm to obtain a matching result of the hand position information of each hand.
5. The method according to claim 4, characterized in that The determining of the hand key point position of each hand in each image frame based on the hand position information of each image frame includes: Extracting the image area indicated by the hand position information of each image frame to obtain each hand image; Each hand image is input into the trained hand feature point regression model to obtain the hand key point position of each hand.
6. The method according to claim 1, characterized in that The determining of the dynamic gesture in the cabin video data stream based on the matching result of the hand position information of each hand in every two consecutive image frames includes: For each hand, when the matching result of the hand position information of the hand in every two consecutive image frames is a matching failure, the hand in the latter image frame between every two consecutive image frames is used as the beginning of a new hand action sequence to construct a hand action sequence; When the matching result of the hand position information of the hand in each two consecutive image frames is a successful match, the hand in each two consecutive image frames is regarded as a continuous part of the same hand action sequence to construct a hand action sequence; Dynamic gestures in the cabin video data stream are generated according to the constructed hand movement sequence.
7. A dynamic gesture tracking device, characterized in that: The device comprises: An image frame acquisition module is used to acquire a vehicle cabin video data stream and capture images of the vehicle cabin video data stream to obtain a plurality of image frames; a first hand position information determination module, configured to trigger hand detection on image frames at predetermined intervals to obtain hand position information for each triggered image frame, wherein the hand position information is coordinate information of a hand circumscribed frame containing each hand; a second hand position information determination module, configured to determine, for each trigger image frame, the hand position information of each non-trigger image frame based on the hand position information of the trigger image frame, wherein the non-trigger image frame is a video frame after the trigger image frame and before the next trigger image frame; a hand position information matching module, configured to match the hand position information of each hand based on the hand position information of every two consecutive image frames to obtain a matching result of the hand position information of each hand, wherein the matching result is a matching success or a matching failure; a dynamic gesture tracking module, configured to determine a dynamic gesture in the cabin video data stream based on a matching result of hand position information of each hand in every two consecutive image frames; The device further includes a location information updating module, which is configured to: After determining the hand position information of each non-trigger image frame based on the hand position information of the trigger image frame, for each trigger image frame, determine the hand key points of each hand in each non-trigger image frame and the confidence level of each hand key point based on the hand position information of each non-trigger image frame; For each non-trigger image frame, determining whether the average confidence value of each hand key point in the non-trigger image frame exceeds a preset threshold; If the average confidence value of the hand key points of each hand in the non-trigger image frame does not exceed the preset threshold, triggering hand detection on the non-trigger image frame to obtain new hand position information of the non-trigger image frame; The hand position information of the non-trigger image frame determined based on the hand position information of the trigger image frame is updated using the new hand position information of the non-trigger image frame.
8. A computer device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the gesture tracking method according to any one of claims 1 to 6 are performed.
9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the gesture tracking method according to any one of claims 1 to 6.
