Manual making action recognition method and device and readable storage medium
By dividing the handmade action video into multiple video clips and extracting the action feature vector, the problem of time-consuming and costly data labeling in the prior art is solved, and efficient recognition of handmade action is achieved.
Patent Information
- Application Number
- CN202411999588.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
The existing manual action recognition method is used to identify through the model, requiring a large amount of data annotation, which results in a long and costly process.
By obtaining the action video of the manual production process, segmenting the video into multiple video clips according to the keyframe, the action feature vector is extracted using the long-term point tracking algorithm and feature description operator, and stored in the action feature library. Finally, it is recognized based on the similarity between the action feature vector of the video clip to be identified and all action feature vectors in the action feature library.
The need for additional data annotation is avoided, and the accurate identification of manual production actions is achieved, reducing the time and cost of data annotation.
Smart Images
Figure CN119920007A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for identifying a hand-made action, and a readable storage medium. Background Art
[0002] In traditional clothing hand-making, craftsmen use their experience and skills to complete various hand-making actions, such as cutting, sewing, matching, etc. However, the identification of these hand-making actions has always been a challenging problem. Traditionally, these skills are passed down through the apprenticeship system, and there is a lack of systematic methods to parse and describe these complex hand-making actions.
[0003] At present, there are many methods in the prior art that can recognize hand-made actions, but the existing methods for recognizing hand-made actions realize recognition through models, which requires a large amount of data annotation, which makes the data annotation process not only time-consuming but also costly. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method, device and readable storage medium for identifying hand-made actions in view of the above-mentioned shortcomings of the prior art, so as to solve the problem that the existing method for identifying hand-made actions realizes recognition through models, which requires a large amount of data annotation, which makes the data annotation process not only time-consuming but also costly.
[0005] In a first aspect, the present invention provides a method for identifying a hand-made action, comprising:
[0006] Get action videos of the hand-crafting process;
[0007] According to key frames in the action video, the action video is divided into a plurality of video clips, wherein the key frames are used to separate different action categories;
[0008] For each video clip in the plurality of video clips, an action feature vector of the video clip is obtained based on a long-term point tracking algorithm and a feature description operator, and the action feature vector is stored in an action feature library;
[0009] The action category of the action to be identified is identified according to the similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library.
[0010] Furthermore, obtaining the motion feature vector of the video clip based on the long-term point tracking algorithm and the feature description operator specifically includes:
[0011] gridding an initial frame in the video clip to obtain a plurality of grid points;
[0012] According to the position of each grid point in the initial frame, a long-term point tracking algorithm is used to obtain the position of each grid point in all subsequent frames of the video segment;
[0013] According to the position of each grid point in the initial frame and all subsequent frames, a moving point gradient network between every two adjacent frames is obtained;
[0014] According to the moving point gradient network between each two adjacent frames, a feature description operator is used to extract the motion feature vector of the video clip.
[0015] Further, obtaining a moving point gradient network between every two adjacent frames according to the position of each grid point in the initial frame and all subsequent frames specifically includes:
[0016] According to the position of each grid point in the initial frame and all subsequent frames, the direction and distance of change of each grid point between each two adjacent frames are calculated to obtain the moving point gradient of each grid point between each two adjacent frames;
[0017] According to the moving point gradients of all the grid points between each two adjacent frames, a moving point gradient network between each two adjacent frames is obtained.
[0018] Furthermore, the moving point gradient network includes a direction moving point gradient network and a distance moving point gradient network, and the motion feature vector of the video clip is extracted using a feature description operator based on the moving point gradient network between each two adjacent frames, specifically including:
[0019] For the directional moving point gradient network and the distance moving point gradient network between each two adjacent frames, calculating the angle interval corresponding to the directional moving point gradient of each grid point in the directional moving point gradient network between the two adjacent frames;
[0020] Calculating the motion feature vector between the two adjacent frames according to the angle interval corresponding to the direction moving point gradient of each grid point in the direction moving point gradient network between the two adjacent frames and the distance moving point gradient of each grid point in the distance moving point gradient network;
[0021] The action feature vector of the video clip is obtained according to the action feature vector between every two adjacent frames.
[0022] Further, the calculating the motion feature vector between the two adjacent frames according to the angle interval corresponding to the direction moving point gradient of each grid point in the direction moving point gradient network between the two adjacent frames and the distance moving point gradient of each grid point in the distance moving point gradient network specifically includes:
[0023] The action feature vector between two adjacent frames is calculated by the following formula:
[0024]
[0025] Among them, V is the motion feature vector between two adjacent frames, m*n is the number of grid points between two adjacent frames, is the angle interval corresponding to the direction moving point gradient of the grid point k between two adjacent frames, u is the angle interval number, The value is 0 or 1, M k is the moving point gradient of the distance of grid point k between two adjacent frames, and ⊙ is the matrix dot product.
[0026] Furthermore, all action feature vectors in the action feature library have corresponding action categories, and the action category of the action to be identified is identified based on the similarity between the action feature vector of the video clip of the action to be identified and all action feature vectors in the action feature library, specifically including:
[0027] Obtaining a video clip of the action to be recognized;
[0028] Calculating the cosine similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library;
[0029] In response to the cosine similarity between the action feature vector of the video clip of the action to be identified and a certain action feature vector among all action feature vectors being greater than a preset threshold, the action category of the action to be identified is identified as the action category corresponding to the certain action feature vector.
[0030] Furthermore, the method further comprises:
[0031] In response to the cosine similarities between the action feature vector of the video clip of the action to be identified and all action feature vectors being less than a preset threshold, sending a manual judgment reminder message to a relevant person so that the relevant person manually judges whether the action category of the action feature vector of the video clip of the action to be identified is a new action category;
[0032] The result of the manual judgment is obtained. If the result of the manual judgment indicates that the action category of the action feature vector of the video clip of the action to be identified is a new action category, the action feature vector corresponding to the new action category is added to the action feature library.
[0033] In a second aspect, the present invention provides a hand-made action recognition device, comprising:
[0034] An acquisition module, used to acquire action videos of the hand-made process;
[0035] A segmentation module, connected to the acquisition module, for segmenting the action video into a plurality of video segments according to key frames in the action video, wherein the key frames are used to separate different action categories;
[0036] A storage module is obtained, connected to the segmentation module, and is used to obtain, for each video segment in the plurality of video segments, an action feature vector of the video segment based on a long-term point tracking algorithm and a feature description operator, and store the result in an action feature library;
[0037] The recognition module is connected to the obtaining and storing module and is used to identify the action category of the action to be recognized based on the similarity between the action feature vector of the video clip of the action to be recognized and all the action feature vectors in the action feature library.
[0038] In a third aspect, the present invention provides a device for identifying handmade actions, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the method for identifying handmade actions described in the first aspect.
[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for recognizing handmade actions described in the first aspect above is implemented.
[0040] The method, device and readable storage medium for identifying hand-made actions provided by the present invention first obtain an action video of a hand-made process; then, according to key frames in the action video, the action video is divided into multiple video clips, wherein the key frames are used to separate different action categories; then, for each video clip in the multiple video clips, an action feature vector of the video clip is obtained based on a long-term point tracking algorithm and a feature description operator, and the result is stored in an action feature library; finally, according to the similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library, the action category of the action to be identified is identified. The present invention divides an action video into multiple video clips representing different action categories accurately, and then uses a long-term point tracking algorithm and a feature description operator to efficiently extract the action feature vector of each video clip and store it in an action feature library. This process ensures the accuracy and representativeness of each action feature vector, and then through the similarity between the action feature vector of the action to be identified and all the action feature vectors in the action feature library, efficient classification of the action to be identified is achieved, avoiding the need for additional data annotation, so that accurate identification of hand-made actions can be achieved without additional data annotation, solving the problem that the existing method for identifying hand-made actions realizes identification through models and requires a large amount of data annotation, which results in a data annotation process that is not only time-consuming but also costly. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is a flow chart of a method for identifying a hand-made action according to Embodiment 1 of the present invention;
[0042] Figure 2 A flowchart of another method for identifying a hand-made action according to an embodiment of the present invention;
[0043] Figure 3 This is a schematic diagram of the structure of a hand-made action recognition device according to Embodiment 2 of the present invention;
[0044] Figure 4 This is a schematic diagram of the structure of a hand-made action recognition device according to Embodiment 3 of the present invention. DETAILED DESCRIPTION
[0045] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0046] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.
[0047] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.
[0048] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.
[0049] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.
[0050] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.
[0051] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.
[0052] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.
[0053] Application Overview
[0054] In recent years, with the development of computer vision, machine learning and artificial intelligence technology, the application of intelligent algorithms in the field of handmade production has attracted increasing attention. Among them, the recognition of clothing handmade movements has become an eye-catching research field. The introduction of this technology can not only improve production efficiency and lower the production threshold, but also promote the digitalization and intelligent development of traditional handmade art. At present, the mainstream technology mainly uses algorithms such as target detection, image classification, posture estimation, and video action segment classification. Object detection technology can be used to identify key objects in images or videos, such as cutting, sewing tools, fabrics, etc. By detecting these key objects, the system can infer the current stage and action of handmade production. Image classification technology can classify images into different categories to identify different actions or processes in handmade production. For example, image classification can be used to distinguish between cutting, sewing, matching and other actions. Posture estimation technology can analyze the posture and movement of the human body in the image to infer the specific steps of handmade production. This is very helpful for capturing the movement of key parts such as hands and body during the production process. For handmade videos, video action segment classification technology can divide the video into different action segments, thereby more finely describing the steps and action changes of handmade production.
[0055] The prior art has some disadvantages, including the following aspects:
[0056] 1. Data labeling is time-consuming and costly: These technologies all use models to recognize manual actions, which requires a large amount of data labeling. Especially in the clothing industry, where clothing styles are frequently changed, algorithm generalization is a difficult problem.
[0057] 2. Poor adaptability: The existing technology has poor adaptability to complex scenes. When there are factors such as lighting changes, occlusion, complex backgrounds, and new clothing styles in the working environment, the detection performance of the existing technology will be affected, resulting in a decrease in accuracy.
[0058] In response to the above technical problems, the present application provides a method, device and readable storage medium for identifying hand-made actions. The method divides the action video into multiple video segments representing different action categories, and then uses a long-term point tracking algorithm and a feature description operator to efficiently extract the action feature vector of each video segment and store it in an action feature library. This process ensures the accuracy and representativeness of each action feature vector. The action to be identified is efficiently classified through the similarity between the action feature vector of the action to be identified and all the action feature vectors in the action feature library, thereby avoiding the need for additional data labeling, so that accurate identification of hand-made actions can be achieved without additional data labeling, thereby at least solving the problem that the existing method for identifying hand-made actions achieves identification through models, which requires a large amount of data labeling, resulting in the data labeling process being not only time-consuming but also costly.
[0059] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.
[0060] Embodiment 1:
[0061] This embodiment provides a method for identifying hand-made actions. Figure 1 As shown, the method includes:
[0062] Step S101: obtaining an action video of the manual production process.
[0063] It should be noted that the manual production process may be a clothing manual production process, or may be a handicraft production process or other manual production process involving relatively fixed mechanical movements.
[0064] Step S102: dividing the action video into a plurality of video segments according to key frames in the action video, wherein the key frames are used to separate different action categories.
[0065] It should be noted that the key frames in the action video of the manual production process may be the initial frames and the end frames of the manual production actions corresponding to different action categories in the action video.
[0066] Specifically, key frames in the action video are extracted, and the action video is segmented into multiple video clips according to the key frames in the action video.
[0067] Step S103: for each video segment in the plurality of video segments, an action feature vector of the video segment is obtained based on a long-term point tracking algorithm and a feature description operator, and the action feature vector is stored in an action feature library.
[0068] In this embodiment, for each video clip, the action feature vector of each video clip is extracted by using a long-term point tracking algorithm and a feature description operator, and the action feature vector of each video clip is stored in an action feature library.
[0069] In an optional embodiment, obtaining the motion feature vector of the video clip based on a long-term point tracking algorithm and a feature description operator specifically includes:
[0070] gridding an initial frame in the video clip to obtain a plurality of grid points;
[0071] According to the position of each grid point in the initial frame, a long-term point tracking algorithm is used to obtain the position of each grid point in all subsequent frames of the video segment;
[0072] According to the position of each grid point in the initial frame and all subsequent frames, a moving point gradient network between every two adjacent frames is obtained;
[0073] According to the moving point gradient network between each two adjacent frames, a feature description operator is used to extract the motion feature vector of the video clip.
[0074] Specifically, the initial frame in the video clip is gridded to obtain m*n grid points, the coordinates of each grid point are recorded, and a long-term point tracking algorithm is applied to the position of each grid point in the initial frame to update the position of each grid point in all subsequent page frames in the video clip.
[0075] Specifically, based on the coordinates of each grid point in the initial frame and subsequent frames, a moving point gradient network between each pair of adjacent frames is obtained, and a feature description operator is used to extract the motion feature vector of the video clip based on the moving point gradient network between each pair of adjacent frames.
[0076] In an optional embodiment, obtaining a moving point gradient network between every two adjacent frames according to the position of each grid point in the initial frame and all subsequent frames specifically includes:
[0077] According to the position of each grid point in the initial frame and all subsequent frames, the direction and distance of change of each grid point between each two adjacent frames are calculated to obtain the moving point gradient of each grid point between each two adjacent frames;
[0078] According to the moving point gradients of all the grid points between each two adjacent frames, a moving point gradient network between each two adjacent frames is obtained.
[0079] Specifically, based on the coordinates of each grid point in the initial frame and all subsequent frames, a moving point gradient network of the grid points between each two adjacent frames is calculated, and specifically, the direction and distance of changes of all grid points between each two frames are calculated to obtain the moving point gradient of each grid point between each two adjacent frames, and further, a moving point gradient network between each two adjacent frames is formed according to the moving point gradients of all grid points between each two adjacent frames.
[0080] It should be noted that if the number of frames of the video clip is not the preset number of frames, the video clip is sampled in the time dimension, and the video clip is fixedly sampled to the preset number of frames, wherein the sampling can be performed before or after calculating the moving point gradient network of the grid points between each two adjacent frames.
[0081] In an optional embodiment, the moving point gradient network includes a direction moving point gradient network and a distance moving point gradient network, and the motion feature vector of the video clip is extracted using a feature description operator based on the moving point gradient network between each two adjacent frames, specifically including:
[0082] For the directional moving point gradient network and the distance moving point gradient network between each two adjacent frames, calculating the angle interval corresponding to the directional moving point gradient of each grid point in the directional moving point gradient network between the two adjacent frames;
[0083] Calculating the motion feature vector between the two adjacent frames according to the angle interval corresponding to the direction moving point gradient of each grid point in the direction moving point gradient network between the two adjacent frames and the distance moving point gradient of each grid point in the distance moving point gradient network;
[0084] The action feature vector of the video clip is obtained according to the action feature vector between every two adjacent frames.
[0085] Specifically, the direction moving point gradient network and the distance moving point gradient network between each two adjacent frames are input into the feature description operator, and the action feature vector is extracted according to the feature description operator. Specifically, the angle range is divided into t equal parts, that is, t intervals, each interval is 360 / t degrees. The angle interval corresponding to the direction moving point gradient of the grid point is calculated. In other words, these grid points are divided into t groups according to their angles, and each group corresponds to an angle interval. Then the action feature vector is calculated, and the action feature vector between the two adjacent frames is calculated by the following formula:
[0086]
[0087] Among them, V is the motion feature vector between two adjacent frames, m*n is the number of grid points between two adjacent frames, is the angle interval corresponding to the direction moving point gradient of the grid point k between two adjacent frames, u is the angle interval number, The value is 0 or 1, M kis the distance moving point gradient of the grid point k between two adjacent frames, and ⊙ is the matrix dot product. That is, for each angle interval, the values of the distance moving point gradient of all grid points in the angle interval are accumulated to obtain t values. This set of values constitutes the histogram array. Finally, the features of all adjacent two frame dimensions of t* are output, flattened into a one-dimensional vector, and normalized to obtain the normalized motion feature vector of the video clip.
[0088] Step S104: identifying the action category of the action to be identified based on the similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library.
[0089] In this embodiment, for a video clip of an action to be identified, the action category of the action to be identified is identified based on the similarity between the action feature vector of the video clip of the action to be identified and all action feature vectors in the action feature library.
[0090] In an optional embodiment, all action feature vectors in the action feature library have corresponding action categories, and identifying the action category of the action to be identified based on the similarity between the action feature vector of the video clip of the action to be identified and all action feature vectors in the action feature library specifically includes:
[0091] Obtaining a video clip of the action to be recognized;
[0092] Calculating the cosine similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library;
[0093] In response to the cosine similarity between the action feature vector of the video clip of the action to be identified and a certain action feature vector among all action feature vectors being greater than a preset threshold, the action category of the action to be identified is identified as the action category corresponding to the certain action feature vector.
[0094] Specifically, a video clip of an action to be identified is obtained and the cosine similarity between an action feature vector of the video clip of the action to be identified and all action feature vectors in an action feature library is calculated. After the cosine similarity between the action feature vector and all action feature vectors in the action feature library is calculated, a threshold p is set. If the cosine similarity between the action feature vector and a certain action feature vector among all action feature vectors is greater than p, it is considered that the action in the action feature library is successfully matched, and the action category of the action to be identified is identified as the action category corresponding to the action feature vector with a cosine similarity greater than p.
[0095] It should be noted that, in response to the cosine similarity between the action feature vector of the video clip of the action to be identified and all action feature vectors being less than a preset threshold, a manual judgment reminder message is sent to relevant personnel to enable the relevant personnel to manually judge whether the action category of the action feature vector of the video clip of the action to be identified is a new action category; the result of the manual judgment is obtained, and if the result of the manual judgment indicates that the action category of the action feature vector of the video clip of the action to be identified is a new action category, the action feature vector corresponding to the new action category is added to the action feature library.
[0096] In a specific embodiment, a method for identifying hand-made actions is provided, which is applied to the field of clothing, such as Figure 2 As shown, the specific steps of the hand-made action recognition method are as follows:
[0097] Step 1: Get a video of the clothing production process;
[0098] Step 2, extracting the key frames (i.e., the start frame and the end frame of each production action) in the video;
[0099] Step 3, video slicing: split the video into video clips according to the key frames in step 2;
[0100] Step 4: For each video slice (i.e., video clip), perform the following steps 5-9 to extract the action feature vector of each video clip. Each feature vector is denoted as v i , where i is the video segment number;
[0101] Step 5, grid the initial frame of the slice to obtain m*n grid points, and record the coordinates of each grid point, where m is preferably 32 and n is preferably 32;
[0102] Step 6: Apply a long-term point tracking algorithm to the position of each grid point in the initial frame, and update the position of each grid point in all subsequent page frames;
[0103] Step 7: According to the result of step 6, the moving point gradient network Δq of the grid points between every two frames is calculated. Specifically, the direction and distance of changes of all grid points between every two frames are calculated, and multiple frames are combined to form the moving point gradient network Δq;
[0104] Step 8: Sample the result of step 7 in the time dimension, and sample the fragments of any frame into c frames (insert frames less than c frames into c frames), where c is preferably 64, and obtain m*n*63 moving point gradient changes, where each grid point k gradient change has a direction change and distance changes Where k is the grid point number, k∈[1,m*n], Represents the displacement of grid point k in the x-direction and y-direction respectively. The displacement of grid point in the x-direction and y-direction is expressed by Calculated, p k is the position of grid point k in the current frame, is the position of the grid point k in the frame after the current frame. The dimension of the result is 2*m*n*63. The direction moving point gradient network is recorded as θ, and the distance moving point gradient network is recorded as M. Finally, c θ and c M are obtained.
[0105] Step 9, feature description operator: Input θ and M in step 8 into the feature description operator, and extract the action feature vector according to the feature description operator. Specifically, first, divide the angle range into t equal parts, t is preferably 18, that is, t intervals, each interval is 360 / t degrees. Calculate θ * ,θ * is the angle interval corresponding to the direction of the moving point gradient of the grid point, where is the angle interval corresponding to the moving point gradient in the direction of grid point k, To round up, we use this to calculate the equally divided intervals that the angle falls into. In other words, these grid points are divided into t groups according to their angles, and each group corresponds to an angle interval. Then, we calculate the motion feature vector V, where u∈[1,t] represents the angle interval number, t is the number of angle intervals, ⊙ is the matrix dot multiplication. That is, in each angle interval, the values of the distance moving point gradient of all grid points in M for the angle interval are accumulated to obtain t values. This set of values constitutes the histogram array. Finally, the features of t*c-1 dimensions are output, flattened into a one-dimensional vector, and normalized to obtain the normalized action feature vector V norm ;
[0106] Step 10. At this point, each video action slice has its own feature vector. Each action is added to the action feature library, and the cosine similarity is used to calculate the similarity between the actions. in are the normalized action feature vectors of video action slices i and j respectively, represents the dot product of vectors, ‖‖ represents the modulus of the vector;
[0107] Step 11, for the newly received video action slice (i.e., the video clip of the action to be identified), after calculating the action feature vector and the cosine similarity, set the threshold p to determine whether the action feature vector of the video clip of the action to be identified is already in the feature vector library. If the similarity is greater than p, it is considered to be successfully matched with the action in the action feature library (i.e., the action category of the action to be identified is identified as the action category corresponding to the action feature vector with a cosine similarity greater than p). Otherwise, manually determine whether it is a new action. If so, mark it and add it to the action feature library.
[0108] It is worth mentioning that the handmade action recognition method proposed in the present invention is based on the grid point long-term point tracking algorithm, and a feature description operator is added to describe the hand movements of workers in the clothing production process as feature vectors, which can effectively separate the worker movements without the need for additional data annotation. In addition, the algorithm of the present invention is not only applicable to the field of clothing, but can also be used for the feature expression and recognition of other mechanical actions and artificial behaviors with a relatively fixed background. The specific steps 2-9 of action disassembly and feature description in the recognition of handmade clothing actions of the present invention include action video segmentation strategy, moving point gradient network sampling strategy, feature description operator design, and the combination and execution order of each module, which achieves the following goals: 1. Reduce the dependence on data annotation. 2. Improve adaptability to complex scenes.
[0109] The method for identifying hand-made actions provided by an embodiment of the present invention first obtains an action video of a hand-made process; then, according to key frames in the action video, the action video is divided into multiple video clips, wherein the key frames are used to separate different action categories; then, for each video clip in the multiple video clips, an action feature vector of the video clip is obtained based on a long-term point tracking algorithm and a feature description operator, and the action feature vector is stored in an action feature library; finally, according to the similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library, the action category of the action to be identified is identified. The present invention divides an action video into multiple video clips representing different action categories accurately, and then uses a long-term point tracking algorithm and a feature description operator to efficiently extract the action feature vector of each video clip and store it in an action feature library. This process ensures the accuracy and representativeness of each action feature vector, and then through the similarity between the action feature vector of the action to be identified and all the action feature vectors in the action feature library, efficient classification of the action to be identified is achieved, avoiding the need for additional data annotation, so that accurate identification of hand-made actions can be achieved without additional data annotation, solving the problem that the existing method for identifying hand-made actions realizes identification through models and requires a large amount of data annotation, which results in a data annotation process that is not only time-consuming but also costly.
[0110] Embodiment 2:
[0111] like Figure 3 As shown, this embodiment provides a hand-made action recognition device, which is used to perform the above-mentioned hand-made action recognition method, including:
[0112] An acquisition module 11 is used to acquire an action video of a manual production process;
[0113] A segmentation module 12, connected to the acquisition module 11, is used to segment the action video into a plurality of video segments according to key frames in the action video, wherein the key frames are used to separate different action categories;
[0114] A storage module 13 is connected to the segmentation module 12 and is used to obtain, for each video segment in the plurality of video segments, an action feature vector of the video segment based on a long-term point tracking algorithm and a feature description operator, and store the obtained action feature vector in an action feature library;
[0115] The recognition module 14 is connected to the obtaining and storing module 13, and is used to identify the action category of the action to be recognized according to the similarity between the action feature vector of the video clip of the action to be recognized and all the action feature vectors in the action feature library.
[0116] Furthermore, the obtaining and storing module 13 specifically includes:
[0117] A gridding unit, used for gridding an initial frame in the video clip to obtain a plurality of grid points;
[0118] A first obtaining unit is used to obtain the position of each grid point in all subsequent frames of the video segment by using a long-term point tracking algorithm according to the position of each grid point in the initial frame;
[0119] A second obtaining unit is used to obtain a moving point gradient network between every two adjacent frames according to the position of each grid point in the initial frame and all subsequent frames;
[0120] The extraction unit is used to extract the motion feature vector of the video clip by using a feature description operator according to the moving point gradient network between each two adjacent frames.
[0121] Furthermore, the second obtaining unit specifically includes:
[0122] A first calculation unit, configured to calculate a direction and a distance of change of each of the grid points between two adjacent frames according to a position of each of the grid points in an initial frame and all subsequent frames, so as to obtain a moving point gradient of each of the grid points between two adjacent frames;
[0123] The third obtaining unit is used to obtain the moving point gradient network between each two adjacent frames according to the moving point gradients of all the grid points between each two adjacent frames.
[0124] Furthermore, the moving point gradient network includes a direction moving point gradient network and a distance moving point gradient network, and the extraction unit specifically includes:
[0125] A second calculation unit is used to calculate the angle interval corresponding to the directional moving point gradient of each grid point in the directional moving point gradient network between the two adjacent frames for the directional moving point gradient network and the distance moving point gradient network between the two adjacent frames;
[0126] A third calculation unit is used to calculate the motion feature vector between the two adjacent frames according to the angle interval corresponding to the direction moving point gradient of each grid point in the direction moving point gradient network between the two adjacent frames and the distance moving point gradient of each grid point in the distance moving point gradient network;
[0127] The fourth obtaining unit is used to obtain the motion feature vector of the video clip according to the motion feature vector between every two adjacent frames.
[0128] Furthermore, the third calculation unit specifically includes:
[0129] The action feature vector between two adjacent frames is calculated by the following formula:
[0130]
[0131] Among them, V is the motion feature vector between two adjacent frames, m*n is the number of grid points between two adjacent frames, is the angle interval corresponding to the direction moving point gradient of the grid point k between two adjacent frames, u is the angle interval number, The value is 0 or 1, M k is the moving point gradient of the distance of grid point k between two adjacent frames, and ⊙ is the matrix dot product.
[0132] Furthermore, all action feature vectors in the action feature library have corresponding action categories, and the recognition module 14 specifically includes:
[0133] An acquisition unit, used for acquiring a video clip of the action to be recognized;
[0134] A fourth calculation unit, used for calculating the cosine similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library;
[0135] The recognition unit is used to identify the action category of the action to be recognized as the action category corresponding to the certain action feature vector in response to the cosine similarity between the action feature vector of the video clip of the action to be recognized and a certain action feature vector among all the action feature vectors being greater than a preset threshold.
[0136] Furthermore, the device also includes:
[0137] A sending module, configured to send a manual judgment reminder message to a relevant person in response to the cosine similarity between the action feature vector of the video clip of the action to be identified and all action feature vectors being less than a preset threshold, so that the relevant person manually judges whether the action category of the action feature vector of the video clip of the action to be identified is a new action category;
[0138] The acquisition and adding module is used to obtain the result of the manual judgment. If the result of the manual judgment indicates that the action category of the action feature vector of the video clip of the action to be identified is a new action category, the action feature vector corresponding to the new action category is added to the action feature library.
[0139] Embodiment 3:
[0140] refer to Figure 4 This embodiment provides a hand-made action recognition device, including a memory 21 and a processor 22, the memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the hand-made action recognition method in Example 1.
[0141] The memory 21 is connected to the processor 22. The memory 21 may be a flash memory, a read-only memory or other memory. The processor 22 may be a central processing unit or a single-chip microcomputer.
[0142] Embodiment 4:
[0143] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for recognizing handmade actions in the above-mentioned embodiment 1 is implemented.
[0144] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.
[0145] In summary, the hand-made action recognition method, device and readable storage medium provided by the embodiments of the present invention first obtain an action video of the hand-made process; then, according to the key frames in the action video, the action video is divided into multiple video clips, wherein the key frames are used to separate different action categories; then, for each video clip in the multiple video clips, the action feature vector of the video clip is obtained based on the long-term point tracking algorithm and the feature description operator, and stored in an action feature library; finally, according to the similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library, the action category of the action to be identified is identified. The present invention divides an action video into multiple video clips representing different action categories accurately, and then uses a long-term point tracking algorithm and a feature description operator to efficiently extract the action feature vector of each video clip and store it in an action feature library. This process ensures the accuracy and representativeness of each action feature vector, and then through the similarity between the action feature vector of the action to be identified and all the action feature vectors in the action feature library, efficient classification of the action to be identified is achieved, avoiding the need for additional data annotation, so that accurate identification of hand-made actions can be achieved without additional data annotation, solving the problem that the existing method for identifying hand-made actions realizes identification through models and requires a large amount of data annotation, which results in a data annotation process that is not only time-consuming but also costly.
[0146] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. A method for identifying hand-made actions, characterized in that: The method comprises: Get action videos of the hand-crafting process; According to key frames in the action video, the action video is divided into a plurality of video clips, wherein the key frames are used to separate different action categories; For each video clip in the plurality of video clips, an action feature vector of the video clip is obtained based on a long-term point tracking algorithm and a feature description operator, and the action feature vector is stored in an action feature library; The action category of the action to be identified is identified according to the similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library.
2. The method according to claim 1, characterized in that The step of obtaining the motion feature vector of the video clip based on the long-term point tracking algorithm and the feature description operator specifically includes: gridding an initial frame in the video clip to obtain a plurality of grid points; According to the position of each grid point in the initial frame, a long-term point tracking algorithm is used to obtain the position of each grid point in all subsequent frames of the video segment; According to the position of each grid point in the initial frame and all subsequent frames, a moving point gradient network between every two adjacent frames is obtained; According to the moving point gradient network between each two adjacent frames, a feature description operator is used to extract the motion feature vector of the video clip.
3. The method according to claim 2, characterized in that The step of obtaining a moving point gradient network between each two adjacent frames according to the position of each grid point in the initial frame and all subsequent frames specifically includes: According to the position of each grid point in the initial frame and all subsequent frames, the direction and distance of change of each grid point between each two adjacent frames are calculated to obtain the moving point gradient of each grid point between each two adjacent frames; According to the moving point gradients of all the grid points between each two adjacent frames, a moving point gradient network between each two adjacent frames is obtained.
4. The method according to claim 2, characterized in that: The moving point gradient network includes a direction moving point gradient network and a distance moving point gradient network. The extracting the motion feature vector of the video clip using a feature description operator based on the moving point gradient network between each two adjacent frames specifically includes: For the directional moving point gradient network and the distance moving point gradient network between each two adjacent frames, calculating the angle interval corresponding to the directional moving point gradient of each grid point in the directional moving point gradient network between the two adjacent frames; Calculating the motion feature vector between the two adjacent frames according to the angle interval corresponding to the direction moving point gradient of each grid point in the direction moving point gradient network between the two adjacent frames and the distance moving point gradient of each grid point in the distance moving point gradient network; The action feature vector of the video clip is obtained according to the action feature vector between every two adjacent frames.
5. The method according to claim 4, characterized in that The step of calculating the motion feature vector between the two adjacent frames according to the angle interval corresponding to the direction moving point gradient of each grid point in the direction moving point gradient network between the two adjacent frames and the distance moving point gradient of each grid point in the distance moving point gradient network specifically includes: The action feature vector between two adjacent frames is calculated by the following formula: Among them, V is the motion feature vector between two adjacent frames, m*n is the number of grid points between two adjacent frames, is the angle interval corresponding to the direction moving point gradient of the grid point k between two adjacent frames, u is the angle interval number, The value is 0 or 1, M k is the moving point gradient of the distance of grid point k between two adjacent frames, and ⊙ is the matrix dot product.
6. The method according to claim 1, characterized in that All action feature vectors in the action feature library have corresponding action categories, and identifying the action category of the action to be identified based on the similarity between the action feature vector of the video clip of the action to be identified and all action feature vectors in the action feature library specifically includes: Obtaining a video clip of the action to be recognized; Calculating the cosine similarity between the action feature vector of the video clip of the action to be identified and all the action feature vectors in the action feature library; In response to the cosine similarity between the action feature vector of the video clip of the action to be identified and a certain action feature vector among all action feature vectors being greater than a preset threshold, the action category of the action to be identified is identified as the action category corresponding to the certain action feature vector.
7. The method according to claim 6, characterized in that The method further comprises: In response to the cosine similarities between the action feature vector of the video clip of the action to be identified and all action feature vectors being less than a preset threshold, sending a manual judgment reminder message to a relevant person so that the relevant person manually judges whether the action category of the action feature vector of the video clip of the action to be identified is a new action category; The result of the manual judgment is obtained. If the result of the manual judgment indicates that the action category of the action feature vector of the video clip of the action to be identified is a new action category, the action feature vector corresponding to the new action category is added to the action feature library.
8. A device for identifying hand-made actions, characterized in that: include: An acquisition module, used to acquire action videos of the hand-made process; A segmentation module, connected to the acquisition module, for segmenting the action video into a plurality of video segments according to key frames in the action video, wherein the key frames are used to separate different action categories; A storage module is obtained, connected to the segmentation module, and is used for obtaining, for each video segment in the plurality of video segments, an action feature vector of the video segment based on a long-term point tracking algorithm and a feature description operator, and storing the result in an action feature library; The recognition module is connected to the obtaining and storing module and is used to identify the action category of the action to be recognized based on the similarity between the action feature vector of the video clip of the action to be recognized and all the action feature vectors in the action feature library.
9. A device for identifying hand-made actions, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the method for recognizing a handmade action as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for recognizing a handmade action according to any one of claims 1 to 7 is implemented.