Hand action recognition method, device and equipment and computer readable storage medium
By detecting the starting position and key point coordinates of the target person in the video and inputting it into the hand motion recognition model, the problems of low efficiency and low accuracy of hand motion recognition in the prior art are solved, and the efficiency and accuracy of automatic recognition of hand motion are achieved.
Patent Information
- Application Number
- CN202311092275.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-28
- Publication Date
- 2025-02-28
AI Technical Summary
In the prior art, hand movement recognition requires a lot of manpower when recognizing videos, which is time-consuming and labor-intensive, has low efficiency and low accuracy.
By obtaining the video of the target to be detected, the starting position and position of each person is detected, the target filter is used to obtain the coordinates of the target person's key point, and the corresponding key point heat map is input to the hand action recognition model to automatically identify the hand movement of the target person.
Automatic recognition of hand movements is realized, which improves recognition efficiency and accuracy, and reduces dependence on human resources.
Smart Images
Figure CN119541033A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of video recognition, and in particular to the technical field of hand motion recognition. Background Art
[0002] At present, when identifying the user's hand movements in the video, the usual operation is manual recognition, that is, the user recognizes based on his own experience. For example, when analyzing the defect detection video of the vehicle body in the factory to confirm the hand movements of the staff, relevant personnel are required to manually analyze the defect detection video frame by frame to determine whether the staff's hand movements are correct. They can also determine whether the staff's defect detection work is in place based on their hand movements. This hand movement recognition method not only requires a lot of manpower and is time-consuming and labor-intensive, but also has low efficiency and low accuracy. Summary of the invention
[0003] The present disclosure provides a hand motion recognition method, device, equipment, storage medium and vehicle.
[0004] According to a first aspect of the present disclosure, a hand motion recognition method is provided. The method comprises:
[0005] Get the target video to be detected;
[0006] Detecting the starting position of the target person in the target video to be detected and the positions of each person in the target video to be detected; wherein the each person includes the target person;
[0007] Inputting the starting position of the target person and the positions of each person into a target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected;
[0008] The key point heat map corresponding to the key point coordinates of the target person in each video frame is input into the hand movement recognition model to obtain the hand movement of the target person, wherein the key point heat map corresponding to the key point coordinates of the target person in each video frame is used to characterize the key point coordinates of the target person in each video frame.
[0009] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the detecting the starting position of the target person in the target video to be detected comprises:
[0010] Confirming the position of each person in the first video frame of the target video to be detected;
[0011] Filter out the positions that are not completely within the preset shielding area from the positions of the personnel;
[0012] The position that does not completely fall into the preset shielding area is determined as the starting position of the target person.
[0013] According to the above aspects and any possible implementation, there is further provided an implementation, wherein the target video to be detected is a defect detection video of a vehicle body-in-white;
[0014] The step of determining a position that is completely outside the preset shielding area as the starting position of the target person includes:
[0015] If the positions of multiple persons among the persons do not completely fall within the position of the preset shielding area, the person closest to the body-in-white is selected from the multiple persons as the target person;
[0016] The position of the target person in the first video frame is determined as the starting position of the target person.
[0017] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the positions of each person in the target video to be detected are obtained by the following steps:
[0018] Calling a pre-trained human key point detection network; wherein the human key point detection network is used to detect the coordinates of the key points of the human body trunk, the key points of the head, the key points of the limbs, and the key points of the palm;
[0019] Input each video frame in the target video to be detected into the human key point detection network to obtain the key point coordinates of each person in each video frame and the human body position of each person;
[0020] The key point coordinates of each person and the body position of each person in each video frame are determined as the position of each person.
[0021] According to the above aspects and any possible implementation, an implementation is further provided, wherein the starting position of the target person and the positions of each person are input into a target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected, including:
[0022] Determine the position of the target person in the first video frame of the target video to be detected as the starting position of the target person;
[0023] Inputting the starting position of the target person and the position of each person in the next video frame of the target video to be detected into the target filter, and obtaining the position of the target person in the next video frame; wherein the next video frame is the next frame adjacent to the video frame where the starting position is located;
[0024] The position of the target person in the second video frame is updated to the starting position of the target person, and the above steps are repeated until the key point coordinates of the target person in each video frame are obtained.
[0025] According to the above aspects and any possible implementation, an implementation is further provided, wherein the starting position of the target person and the position of each person in the next video frame of the target video to be detected are input into a target filter to obtain the position of the target person in the next video frame, including:
[0026] Calculating the degree of overlap between the starting position of the target person and the position of each person in the next video frame;
[0027] Selecting candidate personnel from among the personnel whose repetition degree with the starting position of the target personnel is not less than a set repetition degree;
[0028] The position of the person with the highest degree of overlap with the starting position of the target person is selected from the positions of the candidate persons as the position of the target person in the next video frame.
[0029] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the method further includes:
[0030] If the overlap between the starting position of the target person and the positions of each person in the next video frame is less than the set overlap, it is determined that the search for the target person in the next video frame fails;
[0031] If the search fails twice in succession, the position of the target person in the subsequent video frame is determined as the starting position of the target person, wherein the subsequent video frame is the next video frame adjacent to the video frame where the search for the target person fails.
[0032] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the method further includes:
[0033] Controlling the target sliding window to slide on the key point heat map corresponding to each video frame according to a preset step size;
[0034] After each slide, the key point heat map in the target sliding window is input into the hand motion recognition model to obtain the hand motion of the target person, wherein the size of the target sliding window is n, n is less than the total number of frames of the video frames, and the preset step size is less than n, and n is a positive integer.
[0035] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the method further includes:
[0036] After each sliding, the hand movement corresponding to the key point heat map in the target sliding window is confirmed as the hand movement corresponding to the key point heat map of the first frame in the target sliding window;
[0037] After the last slide, the hand motion corresponding to the key point heat map in the target sliding window is confirmed as the hand motion corresponding to all the key point heat maps in the target sliding window.
[0038] According to a second aspect of the present disclosure, a hand motion recognition device is provided. The device comprises:
[0039] An acquisition module is used to acquire the target video to be detected;
[0040] A detection module, used to detect the starting position of the target person in the target video to be detected and the positions of each person in the target video to be detected; wherein the each person includes the target person;
[0041] A first input module is used to input the starting position of the target person and the positions of each person into a target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected;
[0042] The second input module is used to input the key point heat map corresponding to the key point coordinates of the target person in each video frame into the hand movement recognition model to obtain the hand movement of the target person, wherein the key point heat map corresponding to the key point coordinates of the target person in each video frame is used to characterize the key point coordinates of the target person in each video frame.
[0043] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.
[0044] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.
[0045] According to a fifth aspect of the present disclosure, a vehicle is provided, comprising the hand motion recognition device as described in the second aspect and / or the electronic device as described in the third aspect.
[0046] In the present disclosure, after obtaining the target video to be detected, the starting position of the target person in the target video to be detected and the positions of each person in the target video to be detected can be detected, and then the starting position of the target person and the positions of each person are input into the target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected, and then the key point heat map corresponding to the key point coordinates of the target person in each video frame is automatically input into the hand movement recognition model, and the accurate hand movement of the target person can be obtained. In this way, the hand movement can be automatically identified by using the target filter to avoid the need for manual identification, thereby improving the recognition efficiency and accuracy of hand movements.
[0047] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:
[0049] Figure 1 A flow chart of a hand motion recognition method according to an embodiment of the present disclosure is shown;
[0050] Figure 2 A flowchart of another hand motion recognition method according to an embodiment of the present disclosure is shown;
[0051] Figure 3 A schematic diagram of key points according to an embodiment of the present disclosure is shown;
[0052] Figure 4 A schematic diagram of a target person locking principle according to an embodiment of the present disclosure is shown;
[0053] Figure 5 A schematic diagram of sliding a target sliding window according to an embodiment of the present disclosure is shown;
[0054] Figure 6 A block diagram of a hand motion recognition device according to an embodiment of the present disclosure is shown;
[0055] Figure 7 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0057] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0058] Figure 1 A flow chart of a hand motion recognition method 100 according to an embodiment of the present disclosure is shown. The method 100 may include:
[0059] Step 110, obtaining a target video to be detected;
[0060] The target video to be detected can be a defect detection video of the vehicle body in white. Figure 3 The vehicle shown has only a frame and no wheels. The defect inspection video is a video of staff inspecting defects on the vehicle's body-in-white.
[0061] Step 120, detecting the starting position of the target person in the target video to be detected and the positions of each person in the target video to be detected; wherein the each person includes the target person;
[0062] The target person in the target video to be detected is a person who is located near the vehicle body-in-white and detects defects on the vehicle body-in-white.
[0063] The starting position of the target person can be the position of the target person in the first video frame of the target video to be detected, and of course, it can also include the coordinates of the key points of the target person in the first video frame; the position of each person in the target video to be detected can be the position of each person in each video frame of the target video to be detected, and of course, it can also include the coordinates of the key points of each person in each video frame.
[0064] The starting position of the target person can be detected as follows:
[0065] Confirming the position of each person in the first video frame of the target video to be detected;
[0066] Filter out the positions that are not completely within the preset shielding area from the positions of the personnel;
[0067] The position that does not completely fall into the preset shielding area is determined as the starting position of the target person.
[0068] Step 130, inputting the starting position of the target person and the positions of each person into a target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected;
[0069] The target filter may be a Kalman filter;
[0070] The key point coordinates include: human body torso key point coordinates, head key point coordinates, limb key point coordinates and palm key point coordinates, a total of 19 key point coordinates.
[0071] Step 140, input the key point heat map corresponding to the key point coordinates of the target person in each video frame into the hand movement recognition model to obtain the hand movement of the target person, wherein the key point heat map corresponding to the key point coordinates of the target person in each video frame is used to characterize the key point coordinates of the target person in each video frame.
[0072] Each type of coordinate in the key point heat map is represented by a probability map. Each pixel in the key point heat map is given a probability, which indicates the probability that the point belongs to the key point of the corresponding category. The key point heat map can be equivalent to the probability map. Specifically, the closer the pixel is to the key point, the closer its probability is to 1, and the farther the pixel is from the key point, the closer its probability is to 0. In this way, the probability of the pixel on the key point heat map can be used to confirm which are the key point coordinates. For example, the key point coordinates with a probability greater than 0.4 in the key point heat map are key point coordinates, and the probability less than 0.4 are non-key point coordinates. The key point heat map is generally simulated using a Gaussian function.
[0073] After acquiring the target video to be detected, the starting position of the target person in the target video to be detected and the positions of each person in the target video to be detected can be detected, and then the starting position of the target person and the positions of each person are input into the target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected, and then the key point heat map corresponding to the key point coordinates of the target person in each video frame is automatically input into the hand movement recognition model, and the accurate hand movement of the target person can be obtained. In this way, the hand movement can be automatically identified by using the target filter to avoid the need for manual identification, thereby improving the recognition efficiency and accuracy of hand movements.
[0074] In some embodiments, the detecting the starting position of the target person in the target video to be detected includes:
[0075] Confirming the position of each person in the first video frame of the target video to be detected;
[0076] The position of each person can be represented by a person detection box (such as Figure 4 The person detection frame is detected by using the white box marked with number 1 as shown, that is, the person detection frame is moved in the first video frame to frame each person using the person detection frame, and when each person completely falls into the person detection frame, the position of the person detection frame is the position of each person. For example, when person A falls into the person detection frame, the position of the person detection frame is the position of A, and when person B falls into the person detection frame, the position of the person detection frame is the position of B.
[0077] Of course, the position of each person may also include the coordinates of key points of each person.
[0078] Specifically, the sizes of each video frame in the target video to be detected are the same, and for convenience, the following coordinate system is established, the origin of the coordinate system is the upper left corner of each frame, the horizontal direction to the right of the coordinate origin is the positive horizontal axis of the coordinate system, and the horizontal downward direction of the coordinate origin is the positive vertical axis of the coordinate system. Then, the position of each person is composed of the horizontal and vertical coordinates of the upper left corner and the lower right corner of the person detection frame with reference to the coordinate system.
[0079] Filter out the positions that are not completely within the preset shielding area from the positions of the personnel;
[0080] The preset shielding area is a pre-set area, whose size, color and quantity are all pre-set, such as Figure 4 Black frame shown.
[0081] The preset shielding area is an area where no personnel are selected, that is, the personnel falling into the personnel detection frame of the preset shielding area are not target personnel.
[0082] Of course, the present disclosure may also include a service area for characterizing different parts of a vehicle. Figure 4 Markers are shown with a triangular white frame.
[0083] The position that does not completely fall into the preset shielding area includes a position that completely falls outside the preset shielding area and a position that partially falls inside the preset shielding area and partially falls outside the preset shielding area. If the position of a certain person completely falls outside the preset shielding area, it means that the person is completely outside the preset shielding area. If the position of a certain person partially falls inside the preset shielding area and partially falls outside the preset shielding area, it means that part of the person falls inside the preset shielding area and part of the person falls outside the preset shielding area.
[0084] The position that does not completely fall into the preset shielding area is determined as the starting position of the target person.
[0085] After confirming the positions of each person in the first video frame, the positions that do not completely fall into the preset shielding area can be screened out from the positions of the various persons, that is, the persons who are completely or partially outside the preset shielding area can be screened out, and then the positions that do not completely fall into the preset shielding area can be automatically determined as the starting positions of the target persons.
[0086] In some embodiments, the target video to be detected is a defect detection video of a vehicle body-in-white;
[0087] The vehicle body in white is as follows Figure 4 The body shown has no wheels but only the frame.
[0088] The defect detection video is a video of detecting defects on the vehicle body-in-white.
[0089] The step of determining a position that is completely outside the preset shielding area as the starting position of the target person includes:
[0090] If the positions of multiple persons among the persons do not completely fall within the position of the preset shielding area, the person closest to the body-in-white is selected from the multiple persons as the target person;
[0091] The person closest to the body-in-white is the person with the smallest y coordinate with reference to the above coordinate system.
[0092] The position of the target person in the first video frame is determined as the starting position of the target person, wherein the position of the target person in the first video frame is the position of the person detection frame occupied by the target person in the first video frame.
[0093] If there are multiple persons whose positions do not completely fall within the preset shielding area, the person closest to the body-in-white is selected from the multiple persons as the target person, and then the position of the target person in the first video frame is automatically determined as the starting position of the target person.
[0094] In some embodiments, the position of each person in the target video to be detected is obtained by the following steps:
[0095] Calling a pre-trained human key point detection network; wherein the human key point detection network is used to detect the coordinates of the human body trunk key points, the head key points, the limb key points and the palm key points;
[0096] Key points such as Figure 3 As shown, there are 19 in total.
[0097] Input each video frame in the target video to be detected into the human key point detection network to obtain the key point coordinates of each person in each video frame and the human body position of each person;
[0098] Compared with the human key point detection network in the prior art, the human key point detection network disclosed in the present invention has two more detection points. Among the 17 human key points that can be detected by the human key point detection network in the prior art, the distances between the left and right elbow and wrist key points are defined as d left and d right , the distance from the wrist key point to the extension line of the elbow key point and the wrist key point is δd left and δd right The positions of are newly added key points (where δ∈(0, 1) is a manually defined parameter), that is, the positions of the left and right palm key points are predefined, and a human body key point network capable of detecting the coordinates of 19 key points is obtained.
[0099] The key point coordinates of each person and the body position of each person in each video frame are determined as the position of each person.
[0100] By inputting each video frame in the target video to be detected into the human key point detection network, the key point coordinates of each person in each video frame and the human position of each person can be obtained, and then the key point coordinates of each person in each video frame and the human position of each person are automatically determined as the position of each person.
[0101] In some embodiments, the starting position of the target person and the positions of each person are input into the target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected, including:
[0102] Determine the position of the target person in the first video frame of the target video to be detected as the starting position of the target person;
[0103] The starting position of the target person and the position of each person in the next video frame of the target video to be detected are input into the target filter to obtain the position of the target person in the next video frame; wherein the next video frame is the next frame adjacent to the video frame where the starting position is located; if the video frame where the starting position is located is the second video frame, then the next video frame is the third video frame, and if the video frame where the starting position is located is the third video frame, then the next video frame is the fourth video frame.
[0104] The position of the target person in the second video frame is updated to the starting position of the target person, and the above steps are repeated until the key point coordinates of the target person in each video frame are obtained.
[0105] After determining the starting position of the target person, the starting position of the target person and the position of each person in the next video frame of the target video to be detected can be input into the target filter to obtain the position of the target person in the next video frame, and then the position of the target person in the second video frame is automatically updated to the starting position of the target person, and then the new starting position of the target person and the position in the next video frame are re-input into the target filter to obtain the position of the target person in the next video frame. In this way, the iterative cycle can be continuously performed to track the position of the target person in other video frames except the first video frame in the entire target video to be detected.
[0106] In some embodiments, inputting the starting position of the target person and the position of each person in the next video frame of the target video to be detected into the target filter to obtain the position of the target person in the next video frame includes:
[0107] Calculating the degree of overlap between the starting position of the target person and the position of each person in the next video frame;
[0108] Selecting candidate personnel from among the personnel whose repetition degree with the starting position of the target personnel is not less than a set repetition degree;
[0109] The degree of overlap between the starting position of the target person and the position of each person in the next video frame can be represented by IOU, which is Intersection over Union (a simple measurement standard).
[0110] The position of the person with the highest overlap with the starting position of the target person is selected from the positions of the candidate persons as the position of the target person in the next video frame. The person whose repetition with the starting position of the target person is not less than the set repetition and whose overlap is the highest is the target person.
[0111] By calculating the degree of overlap between the starting position of the target person and the position of each person in the next video frame, candidate persons whose degree of overlap with the starting position of the target person is not less than the set degree of overlap are screened out from the various persons, and the position of the person whose degree of overlap with the starting position of the target person is the highest is automatically screened out from the positions of the candidate persons, the target person in the remaining video frames can be tracked to quickly obtain the position of the target person in the next video frame.
[0112] In some embodiments, the method further comprises:
[0113] If the overlap between the starting position of the target person and the positions of each person in the next video frame is less than the set overlap, it is determined that the search for the target person in the next video frame fails;
[0114] If the search fails twice in a row, the position of the target person in the subsequent video frame is determined as the starting position of the target person, wherein the subsequent video frame is the next video frame adjacent to the video frame in which the search for the target person fails. For example, if the overlap between the position of each person in the second video frame and the third video frame and the starting position is less than the set overlap, it means that the search fails in both the second video frame and the third video frame, and therefore, the position of the target person in the subsequent video frame, i.e., the third video frame, can be determined as the starting position of the target person.
[0115] If the overlap between the starting position of the target person and the positions of each person in the next video frame is less than the set overlap, it is confirmed that the search for the target person in the next video frame has failed, that is, the search for the target person in the next video frame has failed or the target person is not found in the next video frame, and then the number of search failures is recorded. If the search fails twice in a row, the video frame where the search failed can be abandoned, and the position of the target person in the subsequent video frame is determined as the starting position of the target person, that is, the target person is searched again in the subsequent video frame, and the position of the target person in the subsequent video frame is re-used as the starting position of the target person.
[0116] In some embodiments, the method further comprises:
[0117] Controlling the target sliding window to slide on the key point heat map corresponding to each video frame according to a preset step size;
[0118] The key point heat map corresponding to each video frame, that is, each video frame has a key point heat map, so there are as many key point heat maps as there are video frames in the target video to be detected.
[0119] If there are 10 video frames in the target video to be detected and there are 10 key point heat maps, the size of the target sliding window can be the size of 3 video frames, the preset step size can be the size of 1 video frame, and then the target sliding window is moved by the size of 1 video frame each time on the 10 consecutive key point heat maps, such as Figure 5 shown.
[0120] After each slide, the key point heat map in the target sliding window is input into the hand motion recognition model to obtain the hand motion of the target person, wherein the size of the target sliding window is n, n is less than the total number of frames of the video frames, and the preset step size is less than n, and n is a positive integer.
[0121] After creating the target sliding window, the target sliding window can be controlled to slide on the key point heat map corresponding to each video frame according to a preset step size, and then after each sliding, the key point heat map in the target sliding window can be automatically input into the hand motion recognition model to obtain the hand motion of the target person.
[0122] The target sliding window of the present disclosure can perform frame-by-frame analysis and reasoning on the key point heat map corresponding to the target video to be detected of any length, and can achieve real-time action category reasoning and analysis. That is, the present disclosure can obtain the target video to be detected while identifying the hand movements of the target person.
[0123] Accordingly, the sliding window-based approach adopted in the present disclosure can be classified through video context information (i.e., adjacent video frames), can capture actions of any duration, and only requires image classification and annotation for a single frame of image, reducing the cost of manual annotation.
[0124] In some embodiments, the method further comprises:
[0125] After each sliding, the hand movement corresponding to the key point heat map in the target sliding window is confirmed as the hand movement corresponding to the key point heat map of the first frame in the target sliding window;
[0126] Hand actions include but are not limited to "left hand working" (left hand touching, left hand wiping, etc.), "right hand working" (right hand touching, right hand wiping, etc.) and "not in working state".
[0127] After the last slide, the hand motion corresponding to the key point heat map in the target sliding window is confirmed as the hand motion corresponding to all the key point heat maps in the target sliding window.
[0128] After each slide, since there are multiple key point heat maps in the target sliding window, except for the last slide, the hand movements corresponding to the key point heat maps in the identified target sliding window are all the hand movements corresponding to the key point heat maps in the first frame in the target sliding window, and then the cycle continues until the last slide. Since the target sliding window can no longer slide, the hand movements corresponding to the key point heat maps in the target sliding window can be used as the hand movements corresponding to all the key point heat maps in the target sliding window.
[0129] Specifically, if there are 10 video frames in the target video to be detected, there are 10 key point heat maps, the size of the target sliding window can be the size of 3 video frames, and the preset step size can be the size of 1 video frame, then
[0130] In the first sliding, the key point heat map in the target sliding window is the key point heat map of the first frame to the key point heat map of the third frame. Finally, the hand movement corresponding to the key point heat map of the first frame to the key point heat map of the third frame in the target sliding window is the hand movement corresponding to the key point heat map of the first frame, which is also the hand movement of the target person in the first video frame in the target video.
[0131] In the second sliding, the key point heat map in the target sliding window is the key point heat map of the second frame to the key point heat map of the fourth frame. Finally, the hand movements corresponding to the key point heat map of the second frame to the key point heat map of the fourth frame in the target sliding window are the hand movements corresponding to the key point heat map of the second frame, which are also the hand movements of the target person in the target video in the second video frame.
[0132] …
[0133] The eighth slide is the last slide, and the key point heat map in the target sliding window is the key point heat map of the eighth frame to the key point heat map of the tenth frame. Finally, the hand movements corresponding to the key point heat map of the eighth frame to the key point heat map of the tenth frame in the target sliding window are not only the hand movements corresponding to the key point heat map of the eighth frame, but also the hand movements corresponding to the key point heat map of the ninth frame and the hand movements corresponding to the key point heat map of the tenth frame, but also the hand movements of the target person in the target video in the eighth video frame, the hand movements of the target person in the ninth video frame and the hand movements of the target person in the tenth video frame.
[0134] The following will be combined Figures 2 to 5 The technical solution of the present disclosure is further described in detail:
[0135] 1. Call the RGB camera to obtain the staff's Figure 3 The target video to be detected and each video frame of the target video are shot when the white body is subjected to defect detection.
[0136] 2. Input each video frame of the target video into the human key point detection network to obtain the positions of all people in the video frame of the target video and the coordinates of their 19 key points. It should be noted that the number of key points detected by the existing human key point detection network (17-kpts-detnetwork) and related data sets (such as COCO) is 17, and the upper limbs only have shoulder key points, elbow key points, and wrist key points, and it is impossible to obtain accurate palm key points. Therefore, the present disclosure makes fine adjustments on this basis and trains a new network 19-kpts-detnetwork, which can output 2 palm key points based on the original 17 key points. The specific method is as follows:
[0137] A. Preparation stage: First, input the sample image into the 17-kpts-detnetwork for inference to obtain 17 key points. Define the distances between the left and right elbow and wrist key points as d left and d right , the extension lines of the elbow key point and the wrist key point are δd away from the wrist key point left and δd left The positions of the left and right palm key points are taken as the newly added key points (where δ∈(0, 1) is a manually defined parameter), that is, the positions of the left and right palm key points are predefined, and the coordinates of the 19 key points are obtained. After correction, the labeling results of the 19 key points are obtained.
[0138] B. Network structure modification: Modify the output layer of 17-kpts-detnetwork to obtain the new network 19-kpts-detnetwork.
[0139] C. Training and reasoning stage: Each video frame of the target video is input into 19-kpts-detnetwork, and 19 key points of the human body are obtained by reasoning ( Figure 3 ).
[0140] 3. Filter out non-target persons among all detected persons according to certain rules, retain the position of the remaining target persons as the starting position of the Kalman filter (which can be called the starting state), and continue to track the target persons. The specific method is as follows:
[0141] A. Filter non-target personnel according to the prior rules and retain the target personnel. The prior rules diagram is as follows:
[0142] Among them, Figure 4 As shown in the figure, the black frame is the shielding area, among which the white frame marked with the number 1 is the personnel detection frame, and the white frame marked with a triangle is the business area. If the bottom edge of the personnel detection frame is in the black frame, it is a non-target person; if multiple people appear in the non-black frame area at the same time, the detection frame with the highest bottom edge position (that is, the smallest y-axis coordinate value) is taken as the target person.
[0143] B. Get the position of the target person in the first video frame and set it as the starting state of the Kalman filter.
[0144] C. Input the position detection results of all people in the next video frame and input them into the Kalman filter.
[0145] D. The Kalman filter calculates the IOU (degree of overlap) between all personnel positions and the starting state, finds the personnel position corresponding to the maximum IOU greater than the set IOU threshold, and updates the personnel position to the starting state, and then repeats steps C and D until the Kalman filter is used to track the position of the target person in each video frame. Among them, the process of updating the starting state is the process of calculating the Kalman gain and updating the state estimation covariance matrix.
[0146] E. If the IOU of all personnel positions and starting states in a certain video frame is less than the set IOU threshold, it is confirmed that the target person cannot be found in the video frame, that is, the target person search fails, and the number of search failures is recorded. If the target person cannot be found in two consecutive frames, return to step A and reset the target person, that is, the position of the target person in the subsequent video frame of the video frame where the search failed is re-determined as the starting state, and steps B to D are repeated.
[0147] 4. Normalize the key point coordinates of the target person in n adjacent video frames to obtain the key point heat map corresponding to the key point coordinates of each video frame; construct a sliding window with a target sliding window size of n and a step length of m (m <n)。
[0148] 5. Input the target sliding window into the hand action recognition model, determine whether the key action category of the image sequence in the window is "left hand working", "right hand working" or "not in working state", and assign the result to the first frame image in the window, that is, to the key point heat map of the first frame in the target sliding window.
[0149] 6. Move the target sliding window forward with a step size of m, and assign the key action category recognition results frame by frame to the entire video (after the last sliding window, the recognition results are assigned to all key point heat maps in the window).
[0150] Based on the original 17 key points, this paper adds two left and right palm key points, constructs and trains a 19-kpts-detnetwork that can output 19 key points, which to a certain extent solves the occlusion and self-occlusion problems of behavior recognition in factory scenarios.
[0151] The target sliding window can perform frame-by-frame analysis and reasoning on the key point heat map corresponding to the target video to be detected of any length, and can achieve real-time action category reasoning and analysis. That is, the present disclosure can obtain the target video to be detected while identifying the hand movements of the target person.
[0152] Decomposing hand movements into "left hand working", "right hand working" and "not in working state" can be expanded to apply to workstations that require workers to check by touch.
[0153] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.
[0154] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.
[0155] Figure 6 FIG. 6 shows a block diagram of a hand motion recognition device 600 according to an embodiment of the present disclosure. Figure 6 As shown, the device 600 includes:
[0156] The acquisition module 610 is used to acquire the target video to be detected;
[0157] A detection module 620 is used to detect the starting position of the target person in the target video to be detected and the positions of each person in the target video to be detected; wherein the each person includes the target person;
[0158] A first input module 630 is used to input the starting position of the target person and the positions of each person into a target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected;
[0159] The second input module 640 is used to input the key point heat map corresponding to the key point coordinates of the target person in each video frame into the hand motion recognition model to obtain the hand motion of the target person, wherein the key point heat map corresponding to the key point coordinates of the target person in each video frame is used to characterize the key point coordinates of the target person in each video frame.
[0160] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0161] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, including:
[0162] at least one processor; and
[0163] a memory communicatively connected to the at least one processor; wherein,
[0164] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the above method embodiments.
[0165] According to an embodiment of the present disclosure, the present disclosure further provides a vehicle, comprising: the hand motion recognition device as described in the above embodiment or the electronic device as described in the above embodiment.
[0166] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any one of the above method embodiments.
[0167] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0168] The device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0169] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0170] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).
[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0172] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0173] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0175] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0176] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0177] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0178] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A hand motion recognition method, characterized in that: include: Get the target video to be detected; Detecting the starting position of the target person in the target video to be detected and the positions of each person in the target video to be detected; wherein the each person includes the target person; Inputting the starting position of the target person and the positions of each person into a target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected; The key point heat map corresponding to the key point coordinates of the target person in each video frame is input into the hand movement recognition model to obtain the hand movement of the target person, wherein the key point heat map corresponding to the key point coordinates of the target person in each video frame is used to characterize the key point coordinates of the target person in each video frame.
2. The method according to claim 1, characterized in that The detecting the starting position of the target person in the target video to be detected includes: Confirming the position of each person in the first video frame of the target video to be detected; Filter out the positions that are not completely within the preset shielding area from the positions of the personnel; The position that does not completely fall into the preset shielding area is determined as the starting position of the target person.
3. The method according to claim 2, characterized in that The target video to be detected is a defect detection video of the vehicle body in white; The step of determining a position that is completely outside the preset shielding area as the starting position of the target person includes: If the positions of multiple persons among the persons do not completely fall within the position of the preset shielding area, the person closest to the body-in-white is selected from the multiple persons as the target person; The position of the target person in the first video frame is determined as the starting position of the target person.
4. The method according to claim 1, characterized in that: The positions of each person in the target video to be detected are obtained by the following steps: Calling a pre-trained human key point detection network; wherein the human key point detection network is used to detect the coordinates of the key points of the human body trunk, the key points of the head, the key points of the limbs, and the key points of the palm; Input each video frame in the target video to be detected into the human key point detection network to obtain the key point coordinates of each person in each video frame and the human body position of each person; The key point coordinates of each person and the body position of each person in each video frame are determined as the position of each person.
5. The method according to claim 1, characterized in that The starting position of the target person and the positions of each person are input into the target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected, including: Determine the position of the target person in the first video frame of the target video to be detected as the starting position of the target person; Inputting the starting position of the target person and the position of each person in the next video frame of the target video to be detected into the target filter, and obtaining the position of the target person in the next video frame; wherein the next video frame is the next frame adjacent to the video frame where the starting position is located; The position of the target person in the second video frame is updated to the starting position of the target person, and the above steps are repeated until the key point coordinates of the target person in each video frame are obtained.
6. The method according to claim 5, characterized in that The step of inputting the starting position of the target person and the position of each person in the next video frame of the target video to be detected into a target filter to obtain the position of the target person in the next video frame includes: Calculating the degree of overlap between the starting position of the target person and the position of each person in the next video frame; Selecting candidate personnel from among the personnel whose repetition degree with the starting position of the target personnel is not less than a set repetition degree; The position of the person with the highest degree of overlap with the starting position of the target person is selected from the positions of the candidate persons as the position of the target person in the next video frame.
7. The method according to claim 5, characterized in that The method further comprises: If the overlap between the starting position of the target person and the positions of each person in the next video frame is less than the set overlap, it is determined that the search for the target person in the next video frame fails; If the search fails twice in succession, the position of the target person in the subsequent video frame is determined as the starting position of the target person, wherein the subsequent video frame is the next video frame adjacent to the video frame where the search for the target person fails.
8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Controlling the target sliding window to slide on the key point heat map corresponding to each video frame according to a preset step size; After each slide, the key point heat map in the target sliding window is input into the hand motion recognition model to obtain the hand motion of the target person, wherein the size of the target sliding window is n, n is less than the total number of frames of the video frames, and the preset step size is less than n, and n is a positive integer.
9. The method according to claim 8, characterized in that The method further comprises: After each sliding, the hand movement corresponding to the key point heat map in the target sliding window is confirmed as the hand movement corresponding to the key point heat map of the first frame in the target sliding window; After the last slide, the hand motion corresponding to the key point heat map in the target sliding window is confirmed as the hand motion corresponding to all the key point heat maps in the target sliding window.
10. A hand motion recognition device, characterized in that: include: An acquisition module is used to acquire the target video to be detected; A detection module, used to detect the starting position of the target person in the target video to be detected and the positions of each person in the target video to be detected; wherein the each person includes the target person; A first input module is used to input the starting position of the target person and the positions of each person into a target filter to obtain the key point coordinates of the target person in each video frame of the target video to be detected; The second input module is used to input the key point heat map corresponding to the key point coordinates of the target person in each video frame into the hand movement recognition model to obtain the hand movement of the target person, wherein the key point heat map corresponding to the key point coordinates of the target person in each video frame is used to characterize the key point coordinates of the target person in each video frame.
11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.
12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.
13. A vehicle, characterized in that: include: The apparatus as claimed in claim 10, and / or the electronic device as claimed in claim 11, and / or the readable storage medium as claimed in claim 12.