A human behavior recognition method based on deep learning

Abstract

The application discloses a human behavior recognition method based on deep learning, which realizes accurate recognition of human behavior by using an improved GroundedSAM model, wherein the improved GroundedSAM model is improved on the basis of a pseudo-label generation module and a motion feature extraction module of an original GroundedSAM model; the behavior labels in a labeled data set are subjected to semantic extension, an action association library is constructed, and the pseudo-label input of the accurate model is used; the ROI (region of interest) of the motion feature optimization model is combined with optical flow information to extract; a natural language description is generated by using a large language model; finally, a double-channel classifier is used for action prediction and classification. The application can significantly improve the recognition accuracy and effectively reduce the time and calculation cost in the visual understanding process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning behavior recognition, and particularly to a human behavior recognition method based on deep learning. BACKGROUND

[0002] With the explosive growth of video data, the understanding and recognition technology of video content has become one of the important research directions in the field of computer vision. Video action semantic understanding and recognition, as a core task, aims to automatically identify and understand the actions and semantic information of the characters in the video. This technology has wide application prospects in intelligent monitoring, video content analysis, human-computer interaction, virtual reality and other fields.

[0003] Traditional video action recognition methods mainly rely on hand-designed feature extraction and classifiers, such as optical flow, spatio-temporal interest points (STIP), etc. Although these methods have achieved certain results in specific scenarios, they are difficult to cope with complex and variable video content and diversified action categories due to their dependence on manually designed features. In addition, traditional methods often face high computational complexity and poor generalization ability when dealing with large-scale video data.

[0004] In recent years, the rapid development of deep learning technology has brought new breakthroughs to video action recognition. Deep learning methods based on convolutional neural networks (CNN) and recurrent neural networks (RNN) can automatically learn the spatio-temporal features in the video, significantly improving the accuracy and robustness of action recognition. In particular, with the successful application of the Transformer model in natural language processing (NLP), video action recognition methods based on the Transformer have gradually become a research hotspot. These methods can better understand complex actions in videos by capturing long-range dependencies between video frames.

[0005] In existing deep learning methods, video action semantic understanding and recognition still face many challenges. First, the actions in the video often show diversity and complexity, and a single feature extraction method is difficult to fully capture the semantic information of the action. Second, the annotation cost of video data is high, and the existing annotation dataset is limited in size and difficult to cover all possible action categories and scenarios. In addition, how to convert the action information in the video into natural language description for human understanding and subsequent processing is also a problem that needs to be solved. At the same time, the problem of directly recognizing video images cannot be ignored. Each frame in the video may contain a large amount of details and information, but efficient models need to analyze these information complexly, resulting in a significant increase in computational cost and model judgment cost, thereby affecting the real-time performance and accuracy of the model. Therefore, how to balance the comprehensiveness of information capture and the consumption of computing resources is a bottleneck that needs to be broken through in the current video action recognition field. Summary of the Invention

[0006] The purpose of the present invention is to consider the problems of action diversity, complexity, high annotation cost and large computational overhead in video action semantic understanding and recognition, and propose a human action recognition method based on deep learning. This method effectively solves the problems of false detection and missed detection caused by video complexity and action category diversity by introducing pseudo-label generation, motion feature extraction and label-to-text generation, thereby improving the accuracy of video action recognition and effectively reducing costs.

[0007] To achieve the above-mentioned purpose, the technical solution provided by the present invention is: a method for human behavior recognition based on deep learning, which utilizes an improved GroundedSAM model to achieve accurate recognition of human behavior, wherein the improved GroundedSAM model improves the pseudo-label generation module and motion feature extraction module of the original GroundedSAM model; the improvement of the pseudo-label generation module is: based on the original automatic generation of pseudo-labels, the behavior labels in the annotated dataset are semantically expanded to construct an action association library; the improvement of the motion feature extraction module is: a binary mask is initially generated by the frame difference method, and then the optical flow information of the video frame is calculated to obtain the optical flow mask to adjust the model's ROI region of interest;

[0008] The specific implementation of the human behavior recognition method includes the following steps:

[0009] 1) Using the pre-trained language model BERT, we semantically expand the action labels in the annotated human action dataset, generate an extended label set based on the context, and use a semantic embedding model to map the labels in the extended label set into high-dimensional vectors to construct an action association library.

[0010] 2) Extract video frames from the video, generate a preliminary mask through pixel-level difference, and combine optical flow information and connected region analysis to determine the bounding box of the moving area. This is used as the ROI region of interest of the improved GroundedSAM model to accurately determine the main moving area of ​​the character in the video frame;

[0011] 3) Using the video frames extracted in step 2), the RAM model generates preliminary pseudo-labels. Combined with the action association library constructed in step 1), the preliminary pseudo-labels are semantically matched at the vector level with the extended labels in the action association library. The RAG retrieval technique is then used to refine the pseudo-labels, ultimately generating a set of pseudo-labels with higher semantic relevance.

[0012] 4) Using the ROI region of interest and video frame obtained in step 2) to input the trained improved GroundedSAM model to generate a segmentation mask, and semantically match it with the pseudo labels in the pseudo label set to generate a target label set for the video frame;

[0013] 5) Based on the target label set generated in step 4), the pre-trained large language model (LLM) and refined prompt templates are used to convert the target labels in the target label set into natural language descriptions, and then generate accurate and coherent natural language text based on the context;

[0014] 6) In the classification stage, the natural language text generated in step 5) is compared with the behavior labels in the annotated character behavior dataset in step 1), and the similarity and confidence scores are calculated to generate the final classification results.

[0015] Furthermore, in step 1), the behavior labels in the labeled human behavior dataset are semantically expanded using the pre-trained language model BERT. The expansion process generates an extended label set E related to the input label T based on context information through the large language model. The expansion formula is expressed as follows:

[0016] E=B(T,C)

[0017] Where B represents the language model BERT, T is the input behavior label; C is the context information used to guide the generated relevance; E is the generated extended label set {E1, E2, ..., E i ,...,E n}, where E i is the i-th extended tag, i = 1, 2, 3..., n;

[0018] The extended label set E is mapped to a high-dimensional vector space through the pre-trained semantic embedding model F to obtain the vector representation of each label:

[0019] v i =F(E i )

[0020] Where, v i is the i-th label vector

[0021] All embedded vectors are stored in the action association library and used as the matching basis for subsequent pseudo-label generation. The action association library is defined as the vector set V:

[0022] V={v1,v2,...,v n}.

[0023] Furthermore, in step 2), the ROI area is dynamically adjusted by using the pixel difference of the video frames extracted from the video and combining the changes in the optical flow information. The specific steps are as follows: From the original video data, continuous video frames F are extracted according to the frame extraction rule of Q frames per second. t , where t represents the time index, and the frame extraction rule is expressed as:

[0024]

[0025] Where V′ is the input video, Δ is the frame interval, and F t is the extracted video frame;

[0026] For the extracted video frame F t , perform pixel-level difference processing on adjacent frame images, calculate the frame difference of each pixel position (x, y) in the image, that is, the change in the pixel value at this position in the i+1th video frame; by averaging the frame differences and setting the intensity threshold θ, the difference result is initially generated into a binary mask M(x, y):

[0027]

[0028] In the formula, x and y represent the x-axis and y-axis values ​​of the video frame respectively, and D i+1 (x, y) is the frame difference of the i+1th video frame at position (x, y), I i (x,y) is the pixel value of the i-th video frame, I i+1 (x,y) is the pixel value of the i+1th video frame;

[0029] By analyzing the motion direction and speed of each pixel in the video frame, the optical flow information of each pixel is obtained. Specifically, the optical flow constraint equation is used: x (x,y,t)u(x,y,t)+I y (x,y,t)v(x,y,t)+I t (x,y,t)=0, and the optical flow vector u(x,y,t) and v(x,y,t) of each pixel are obtained, where u(x,y,t) represents the velocity component of the optical flow in the horizontal direction, and v(x,y,t) represents the velocity component of the optical flow in the vertical direction. x (x,y,t) and I y (x, y, t) are the gradients of the video frame in the x and y directions, respectively, indicating the change in image intensity; I t (x, y, t) is the temporal variation of the video frame intensity, representing the intensity difference between consecutive frames;

[0030] Then calculate the optical flow vector modulus L as:

[0031]

[0032] The optical flow vector modulus L represents the displacement of the pixel in unit time;

[0033] In order to combine the pixel information and the optical flow information to more accurately obtain the specific range of the motion area, the optical flow mask M of the key frame is calculated based on the obtained binary mask M(x, y) and the optical flow vector modulus L. Flow (x,y), the specific formula is as follows:

[0034]

[0035] Where, is the initial threshold;

[0036] Optical flow mask M Flow (x, y) performs connected region analysis and calculates the coordinates of the minimum bounding rectangle of the region. The coordinates of the minimum bounding rectangle are used as the ROI region of interest of the improved GroundedSAM model.

[0037] Further, in step 3), the video frame F extracted in step 2) is t Input into the RAM model and output the object and action label set of the corresponding frame, where L i Defined as the set of preliminary pseudo labels;

[0038] In order to optimize the semantic accuracy of the preliminary pseudo-label set, it is semantically matched with the extended label set in the action association library; the semantic similarity calculation is performed by calculating the preliminary pseudo-label set L i The embedding vector u i The label vector v in the action association library j The cosine similarity of the two is used to evaluate the semantic relevance between them. The similarity formula is:

[0039]

[0040] Where S ij is the cosine similarity between the two;

[0041] The RAG retrieval technology is used to retrieve the k extended labels with the highest similarity from the action association library to generate the candidate pseudo label set W i :

[0042] W i =Top-k({S ij})

[0043] From the candidate pseudo-label set W i Select the most semantically relevant labels as the final pseudo-label set P i :

[0044] Pi =Select(W i )

[0045] The final pseudo-label set P i With the preliminary pseudo-label set L i Maintain semantic consistency and optimize label accuracy.

[0046] Furthermore, in step 4), the ROI region of interest and the video frame generated in step 2) are used as the image input of the improved GroundedSAM model to generate a segmentation mask. Subsequently, the GroundedSAM model fuses the generated segmentation mask with the pseudo-label generated in step 3) to perform text features and image features, and performs target detection based on the fused features to generate a target label set A = {a1, a2, ..., a n'}, where a n' is the n'th target label.

[0047] Furthermore, in step 5), the LLM model and the designed Prompt template are used to realize the conversion from target label to natural language generation. The specific steps are as follows:

[0048] Prompt template design: Design prompt templates that adapt to different scenarios and behaviors. i , ensuring that the target label set A can be effectively converted into natural language description;

[0049] Using the pre-trained large language model LLM, according to the designed Prompt template O i and the target label set A, generate the natural language text S={s1,s2,...,s i' ,...,s m},satisfy:

[0050] s i' =LLM(O i' ,T i' )

[0051] Where s i' represents the i'th natural language text, O i' Indicates the i'th Prompt template, T i' Represents the i'th behavior label.

[0052] Further, in step 6), the following specific steps are implemented:

[0053] Dual-channel input processing: The natural language text S and behavior label T generated in step 5) are input into the two channels of the dual-channel classifier respectively, and independent feature extraction is performed to obtain feature vectors f1 and f2;

[0054] Feature vector comparison: The extracted feature vectors f1 and f2 are used to calculate the similarity and consistency between the text and the true label. The similarity calculation formula is as follows:

[0055]

[0056] Confidence score generation: Based on the comparison results of the feature vectors, a confidence score is generated for each behavior recognition result to reflect the reliability of the prediction. The confidence score C′ is calculated using the following formula:

[0057] C′=(Similarity)

[0058] Where, σ is the Sigmoid function;

[0059] Behavior category prediction: Based on the confidence score and similarity results, the natural language text S is classified into the closest behavior category. The classification rules are as follows:

[0060]

[0061] Where C i* Represents the confidence score of the i*th behavior category, and predictedBehavior represents the closest behavior category obtained by classification.

[0062] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0063] 1. This paper introduces the pre-trained language model BERT to semantically expand behavior labels and builds an action association library in conjunction with a semantic embedding model, significantly improving the semantic coverage of labels. By combining the semantic matching between the preliminary pseudo-labels generated by the RAM model and the action association library, and using the RAG retrieval mechanism to accurately correct the pseudo-labels, the system effectively improves its semantic understanding of diverse and complex human behaviors, enhancing robustness and recognition accuracy.

[0064] 2. This invention effectively reduces the burden of manual annotation and significantly saves data preparation time by generating pseudo-labels and expanding label semantics. This invention combines optical flow information with connected region analysis to dynamically identify key motion regions as the ROI of GroundedSAM, enhancing the accuracy of target detection.

[0065] 3. Compared with traditional methods that perform redundant calculations on the entire video, this invention significantly reduces the processing requirements of irrelevant areas through optical flow and ROI area positioning. At the same time, the improved GroundedSAM model and segmentation mask mechanism further reduce the computational overhead. The pseudo-label mechanism and natural language generation significantly reduce the dependence on large-scale manually labeled data. Finally, the similarity and confidence scores are combined in the classification stage to ensure the accuracy of the results while reducing the computing power cost during training and inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Flowchart of the method of the present invention.

[0067] Figure 2 Generate the final pseudo-label view based on the action association library and the GroundedSAM model segmentation mask.

[0068] Figure 3 Flowchart for optimizing the region of interest (ROI).

[0069] Figure 4 This is the dual-channel feature matching and evaluation flow chart. DETAILED DESCRIPTION

[0070] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0071] like Figures 1 to 4 As shown, this embodiment discloses a method for human behavior recognition based on deep learning. The method uses an improved GroundedSAM model to achieve accurate recognition of human behavior, wherein the improved GroundedSAM model improves the pseudo-label generation module and motion feature extraction module of the original GroundedSAM model; the improvement of the pseudo-label generation module is: based on the original automatic generation of pseudo-labels, the behavior labels in the annotated data set are semantically expanded to construct an action association library; the improvement of the motion feature extraction module is: a binary mask is initially generated by the frame difference method, and then the optical flow information of the video frame is calculated to obtain the optical flow mask to adjust the model's ROI region of interest;

[0072] The specific implementation of the human behavior recognition method includes the following steps:

[0073] 1) Using the pre-trained language model BERT, we semantically expand the behavior labels in the annotated human behavior dataset, generate an extended label set based on the context, and use a semantic embedding model to map the labels in the extended label set into high-dimensional vectors to construct an action association library. The details are as follows:

[0074] The behavior label in the annotated human behavior data set is semantically expanded by using a pre-trained language model BERT. The expansion process generates an expansion label set E related to the input label T based on the context information by a large language model. The expansion formula is as follows:

[0075] E=B(T,C)

[0076] In the formula, B represents the language model BERT, T is the input behavior label, C is the context information for guiding the relevance of the generated, and E is the generated expansion label set {E1, E2,..., En}, where Ei is the ith expansion label, i=1, 2, 3,..., n. i ,...,E n} where E i is the ith expansion label, i=1, 2, 3,..., n.

[0077] The expansion label set E is mapped to a high-dimensional vector space by a pre-trained semantic embedding model F to obtain the vector representation of each label:

[0078] v i =F(E i )

[0079] In the formula, v i is the ith label vector

[0080] All embedded vectors are stored in the action association library, which is used as the basis for matching subsequent pseudo-label generation. The action association library is defined as a vector set V:

[0081] V={v1,v2,...,v n}

[0082] 2) Extract video frames from the video, generate preliminary masks through pixel-level difference, and determine the bounding box of the motion area by combining optical flow information and connected region analysis, which is the ROI region of interest of the improved GroundedSAM model, to accurately determine the main motion area of the person in the video frame, as follows:

[0083] By extracting video frames from the video, using pixel difference of the video frames, combining changes of optical flow information, and dynamically adjusting the ROI region, the specific steps are as follows: from the original video data, extract continuous video frames F t , where t represents the time index, and the frame extraction rule is represented as:

[0084]

[0085] In the formula, V' is the input video, Delta is the frame extraction interval, and F t is the extracted video frame.

[0086] For the extracted video frames Ft , perform pixel-level difference processing on adjacent frame images, calculate the frame difference of each pixel position (x, y) in the image, that is, the change in the pixel value at this position in the i+1th video frame; by averaging the frame differences and setting the intensity threshold θ, the difference result is initially generated into a binary mask M(x, y):

[0087]

[0088] In the formula, x and y represent the x-axis and y-axis values ​​of the video frame respectively, and D i+1 (x, y) is the frame difference of the i+1th video frame at position (x, y), I i (x,y) is the pixel value of the i-th video frame, I i+1 (x,y) is the pixel value of the i+1th video frame;

[0089] By analyzing the motion direction and speed of each pixel in the video frame, the optical flow information of each pixel is obtained. Specifically, the optical flow constraint equation is used: x (x,y,t)u(x,y,t)+I y (x,y,t)v(x,y,t)+I t (x,y,t)=0, and the optical flow vector u(x,y,t) and v(x,y,t) of each pixel are obtained, where u(x,y,t) represents the velocity component of the optical flow in the horizontal direction, and v(x,y,t) represents the velocity component of the optical flow in the vertical direction. x (x,y,t) and I y (x, y, t) are the gradients of the video frame in the x and y directions, respectively, indicating the change in image intensity; I t (x, y, t) is the temporal variation of the video frame intensity, representing the intensity difference between consecutive frames;

[0090] Then calculate the optical flow vector modulus L as:

[0091]

[0092] The optical flow vector modulus L represents the displacement of the pixel in unit time;

[0093] In order to combine the pixel information and the optical flow information to more accurately obtain the specific range of the motion area, the optical flow mask M of the key frame is calculated based on the obtained binary mask M(x, y) and the optical flow vector modulus L. Flow (x,y), the specific formula is as follows:

[0094]

[0095] Where, is the initial threshold;

[0096] Optical flow mask M Flow (x, y) performs connected region analysis and calculates the coordinates of the minimum bounding rectangle of the region. The coordinates of the minimum bounding rectangle are used as the ROI region of interest of the improved GroundedSAM model.

[0097] 3) Using the video frames extracted in step 2), the RAM model generates preliminary pseudo-labels. Combined with the action association library constructed in step 1), the preliminary pseudo-labels are semantically matched at the vector level with the extended labels in the action association library. Using RAG retrieval technology, the pseudo-labels are refined, ultimately generating a set of pseudo-labels with higher semantic relevance. Specifically, the following is the code:

[0098] The video frame F extracted in step 2) t Input into the RAM model and output the object and action label set of the corresponding frame, where L i Defined as the set of preliminary pseudo labels;

[0099] In order to optimize the semantic accuracy of the preliminary pseudo-label set, it is semantically matched with the extended label set in the action association library; the semantic similarity calculation is performed by calculating the preliminary pseudo-label set L i The embedding vector u i The label vector v in the action association library j The cosine similarity of the two is used to evaluate the semantic relevance between them. The similarity formula is:

[0100]

[0101] Where S ij is the cosine similarity between the two;

[0102] The RAG retrieval technology is used to retrieve the k extended labels with the highest similarity from the action association library to generate the candidate pseudo label set W i :

[0103] W i =Top-k({S ij})

[0104] From the candidate pseudo-label set W i Select the most semantically relevant labels as the final pseudo-label set P i :

[0105] P i =Select(W i )

[0106] The final pseudo-label set P i With the preliminary pseudo-label set L i Maintain semantic consistency and optimize label accuracy.

[0107] 4) The ROI region of interest and the video frame generated in step 2) are used as the image input of the improved GroundedSAM model to generate a segmentation mask. Subsequently, the GroundedSAM model fuses the generated segmentation mask with the pseudo-label generated in step 3) to perform text features and image features, and performs target detection based on the fused features to generate a target label set A = {a1, a2, ..., a n'}, where a n' is the n'th target label.

[0108] 5) Based on the target label set generated in step 4), the pre-trained large language model (LLM) and refined prompt template are used to convert the target labels in the target label set into natural language descriptions. This is then combined with the context to generate accurate and coherent natural language text, as follows:

[0109] Using the LLM model and designing the Prompt template, we can achieve the conversion from target label to natural language generation. The specific steps are as follows:

[0110] Prompt template design: Design prompt templates that adapt to different scenarios and behaviors. i , ensuring that the target label set A can be effectively converted into natural language description;

[0111] Using the pre-trained large language model LLM, according to the designed Prompt template O i and the target label set A, generate the natural language text S={s1,s2,...,s i' ,...,s m},satisfy:

[0112] s i' =LLM(O i' ,T i' )

[0113] Where s i' represents the i'th natural language text, O i' Indicates the i'th Prompt template, T i' Represents the i'th behavior label.

[0114] 6) In the classification phase, the natural language text generated in step 5) is compared with the behavior labels in the annotated character behavior dataset in step 1), and the similarity and confidence scores are calculated to generate the final classification results. This is achieved through the following specific steps:

[0115] Dual-channel input processing: the natural language text S generated in step 5) and the behavior label T are respectively input into two channels of a dual-channel classifier for independent feature extraction to obtain feature vectors f1 and f2;

[0116] Feature vector comparison: the extracted feature vectors f1 and f2 are compared by calculating the similarity Similarity to evaluate the similarity and consistency between the text and the true label, and the similarity calculation formula is as follows:

[0117]

[0118] Confidence score generation: according to the comparison result of the feature vectors, the confidence score of each behavior recognition result is generated to reflect the reliability of the prediction, and the confidence score C' is calculated by the following formula:

[0119] C'= σ (Similarity)

[0120] In the formula, σ is a Sigmoid function;

[0121] Behavior category prediction: according to the confidence score and the similarity result, the natural language text S is classified into the closest behavior category, and the classification rule is as follows:

[0122]

[0123] In the formula, C i* represents the confidence score of the i*th behavior category, and predictedBehavior represents the closest behavior category obtained by classification.

[0124] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods, which are all included in the protection scope of the present application.

Claims

1. A method for character behavior recognition based on deep learning, characterized in that: This method uses an improved GroundedSAM model to achieve accurate recognition of human behavior. The improved GroundedSAM model improves the pseudo-label generation module and motion feature extraction module of the original GroundedSAM model. The improvement of the pseudo-label generation module is as follows: on the basis of the original automatic generation of pseudo-labels, the behavior labels in the annotated dataset are semantically expanded to build an action association library; the improvement of the motion feature extraction module is as follows: a binary mask is initially generated by the frame difference method, and then the optical flow information of the video frame is calculated to obtain the optical flow mask to adjust the model's ROI region of interest. The specific implementation of the human behavior recognition method includes the following steps: 1) Using the pre-trained BERT language model, we semantically expand the action labels in the annotated human action dataset, generate an extended label set based on the context, and use a semantic embedding model to map the labels in the extended label set into high-dimensional vectors to construct an action association library. 2) Extract video frames from the video, generate a preliminary mask through pixel-level difference, and combine optical flow information and connected region analysis to determine the bounding box of the moving area. This is used as the region of interest (ROI) of the improved GroundedSAM model to accurately determine the main moving area of ​​the character in the video frame; 3) Using the video frames extracted in step 2), the RAM model generates preliminary pseudo-labels. Combined with the action association library constructed in step 1), the preliminary pseudo-labels are semantically matched at the vector level with the extended labels in the action association library. Using RAG retrieval technology, the pseudo-labels are refined, ultimately generating a set of pseudo-labels with higher semantic relevance. 4) Use the ROI region of interest and video frame obtained in step 2) to input the trained improved GroundedSAM model to generate a segmentation mask, and perform semantic matching with the pseudo labels in the pseudo label set to generate the target label set of the video frame; 5) Based on the target label set generated in step 4), the pre-trained large language model (LLM) and refined prompt templates are used to convert the target labels in the target label set into natural language descriptions, and then generate coherent natural language text based on the context. 6) In the classification stage, the natural language text generated in step 5) is compared with the behavior labels in the annotated character behavior dataset in step 1), and the similarity and confidence scores are calculated to generate the final classification results.

2. The method for character behavior recognition based on deep learning according to claim 1, characterized in that: In step 1), the behavior labels in the labeled character behavior dataset are semantically expanded using the pre-trained language model BERT. The expansion process generates and inputs labels based on context information through a large language model. Related extension tag collection , the expanded formula is as follows: ； Where, Represents the language model BERT, is the input behavior label; is contextual information, used to guide the generated relevance; A collection of generated extension tags ,in For the Extension tags, ; Will expand the tag collection Through pre-trained semantic embedding model Mapping to a high-dimensional vector space, we get the vector representation of each label: ； Where, For the label vector All embedded vectors are stored in the action association library and used as the matching basis for subsequent pseudo-label generation. The action association library is defined as a vector set : 。 3. The method for character behavior recognition based on deep learning according to claim 2, characterized in that: In step 2), the ROI area is dynamically adjusted by using the pixel difference of the video frames extracted from the video and combining the changes in the optical flow information. The specific steps are as follows: Continuous video frames are extracted from the original video data according to the frame extraction rule of Q frames per second. ,in Represents the time index, and the frame extraction rule is expressed as: ； Where, For input video, is the frame extraction interval, is the extracted video frame; The extracted video frames , perform pixel-level difference processing on adjacent frame images and calculate the position of each pixel in the image The frame difference, that is, The pixel value changes of the video frames at this position; by averaging the frame differences and setting the intensity threshold , the difference result is initially generated into a binary mask : ； ； Where, 、 Represents the video frames 、 Axis direction value, For the Video frames at position The frame difference at It is The pixel value of a video frame, It is The pixel value of each video frame; By analyzing the motion direction and speed of each pixel in the video frame, the optical flow information of each pixel is obtained. Specifically, the optical flow constraint equation is used: , solve the optical flow vector of each pixel and ,in Represents the velocity component of the optical flow in the horizontal direction, Represents the velocity component of the optical flow in the vertical direction, and The video frames are and The gradient in the direction indicates the change in image intensity; is the temporal variation of video frame intensity, representing the intensity difference between consecutive frames; Then calculate the optical flow vector modulus for: ； Optical flow vector modulus Indicates the displacement of a pixel per unit time; In order to combine the pixel information and optical flow information to obtain the specific range of the motion area, based on the obtained binary mask and the optical flow vector modulus , calculate the optical flow mask of the key frame , the specific formula is as follows: ； Where, is the initial threshold; Optical flow mask Perform connected region analysis and calculate the coordinates of the minimum circumscribed rectangle of the region; The coordinates of the minimum bounding rectangle are used as the ROI region of interest of the improved GroundedSAM model.

4. The method for human behavior recognition based on deep learning according to claim 3, wherein: In step 3), the video frame extracted in step 2) is Input into the RAM model and output the object and action label set of the corresponding frame, where Defined as the set of preliminary pseudo labels; In order to optimize the semantic accuracy of the preliminary pseudo-label set, it is semantically matched with the extended label set in the action association library; the semantic similarity calculation is performed by calculating the preliminary pseudo-label set Embedding vector of Label vectors in the action association library The cosine similarity of the two is used to evaluate the semantic relevance between them. The similarity formula is: ； Where, is the cosine similarity between the two; Use RAG retrieval technology to retrieve the most similar action from the action association database Extended labels to generate a set of candidate pseudo labels : ； From the candidate pseudo-label set Select the most semantically relevant labels as the final pseudo-label set : ； Final pseudo-label set With the preliminary pseudo-label set Maintain semantic consistency and optimize label accuracy.

5. The method for human behavior recognition based on deep learning according to claim 4, characterized in that: In step 4), the ROI region of interest and video frame generated in step 2) are used as image inputs of the improved GroundedSAM model to generate a segmentation mask. Subsequently, the GroundedSAM model fuses the generated segmentation mask with the pseudo-label generated in step 3) to perform text features and image features, and performs target detection based on the fused features to generate a target label set. ,in For the target labels.

6. The method for human behavior recognition based on deep learning according to claim 5, characterized in that: In step 5), the LLM model and the Prompt template are used to convert the target label to natural language generation. The specific steps are as follows: Prompt template design: Design prompt templates that adapt to different scenarios and behaviors , ensuring that the target label set A can be effectively converted into natural language description; Using the pre-trained large language model LLM, according to the designed Prompt template And the target label set A, generate natural language text of the character behavior in the video ,satisfy: ； Where, Indicates the natural language text, Indicates the A Prompt template, Indicates the Behavior tags.

7. The method for human behavior recognition based on deep learning according to claim 6, characterized in that: In step 6), this is achieved through the following specific steps: Dual-channel input processing: the natural language text generated in step 5) and behavioral labels Input the two channels of the dual-channel classifier separately, perform independent feature extraction, and obtain the feature vector and ; Feature vector comparison: extracted feature vectors and By calculating the similarity , evaluate the similarity and consistency between the text and the true label, the similarity calculation formula is as follows: ； Confidence score generation: Based on the comparison results of the feature vectors, a confidence score is generated for each behavior recognition result to reflect the reliability of the prediction. Calculated by the following formula: ； Where, is the Sigmoid function; Behavior category prediction: Based on the confidence score and similarity results, natural language text Classify into the closest behavior category, the classification rules are as follows: ； Where, Represents the confidence score of the i*th behavior category, and predictedBehavior represents the closest behavior category obtained by classification.

Images