Object behavior recognition method and device, model training method and device and electronic equipment

Abstract

The embodiment of the invention provides an object behavior recognition method and device, a model training method and device and electronic equipment, and relates to the technical field of video processing. The object behavior recognition method comprises the steps of obtaining a to-be-recognized video; inputting the to-be-recognized video into a video feature extraction network in a pre-trained object behavior recognition model to obtain global time domain features and global spatial features of the to-be-recognized video as to-be-processed features; extracting an image feature of an image area occupied by a to-be-recognized object in the to-be-recognized video by using a first feature extraction network, and taking the image feature as an appearance feature of the to-be-recognized object; performing feature fusion on the appearance features of the to-be-identified object and the to-be-processed features to obtain object form adaptive features; and deep feature extraction is performed on the to-be-recognized features based on the first large language model to obtain a target behavior recognition result of the to-be-recognized object, so that interference caused by drawing style differences is reduced, and the universality of the object behavior recognition model is improved.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to an object behavior recognition method, model training method, apparatus, and electronic device. Background Technology

[0002] In the field of video processing technology, an object behavior recognition model with an initial structure can be trained using sample videos containing sample objects and sample labels indicating the behavior of the sample objects until the model converges, resulting in a trained object behavior recognition model. Then, the trained object behavior recognition model is used to identify the behavior of the object to be identified in the video. For example, the object behavior recognition model can include C3D (Convolutional 3 Dimensions) and LLM (Large Language Model). After inputting the video to be identified into the object behavior recognition model, C3D is used to extract video features; based on Q-former (Query-Former) technology, the video features are converted into text features; and LLM is used to perform deep feature extraction on the converted text features to obtain the behavior of the object to be identified.

[0003] Since object behavior recognition models are trained on sample videos, the accuracy of the recognition results is high when the style of the object to be recognized is similar to that of the sample object. However, if the style of the object to be recognized differs significantly from that of the sample character, the accuracy of the recognition results may be lower. In other words, the general applicability of existing object behavior recognition models is limited. Summary of the Invention

[0004] The purpose of this invention is to provide an object behavior recognition method, a model training method, a device, and an electronic device to improve the versatility of object behavior recognition models. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, an object behavior recognition method is provided, the method comprising: acquiring a video to be recognized; inputting the video to be recognized into a video feature extraction network in a pre-trained object behavior recognition model to obtain global temporal features and global spatial features of the video to be recognized as features to be processed; wherein the object behavior recognition model further comprises a first feature extraction network and a first large language model; using the first feature extraction network to extract image features of the image region occupied by the object to be recognized in the video to be recognized as appearance features of the object to be recognized; performing feature fusion on the appearance features of the object to be recognized and the features to be processed to obtain object morphology adaptation features; and performing deep feature extraction on the features to be recognized based on the first large language model to obtain the target behavior recognition result of the object to be recognized; wherein the features to be recognized are obtained based on the object morphology adaptation features.

[0006] In a second aspect of this invention, a model training method is also provided, the method comprising: acquiring a sample video and sample labels indicating the sample behavior of sample objects in the sample video; inputting the sample video into a video feature extraction network in an initial structure object behavior recognition model to obtain global temporal features and global spatial features of the sample video as first sample features; wherein, the object behavior recognition model further comprises a first feature extraction network and a first large language model; using the first feature extraction network to extract image features of the image region occupied by the sample object in the sample video as appearance features of the sample object; performing feature fusion on the appearance features of the sample object and the first sample features to obtain sample morphology adaptation features; performing deep feature extraction on the second sample features based on the first large language model to obtain a predicted behavior recognition result of the sample object; wherein, the second sample features are obtained based on the sample morphology adaptation features; adjusting the model parameters of the initial structure object behavior recognition model based on the difference between the predicted behavior recognition result and the sample labels until the model converges to obtain a trained object behavior recognition model.

[0007] In a third aspect of the present invention, an object behavior recognition device is provided, the device comprising:

[0008] A first acquisition module is used to acquire a video to be identified; a first feature extraction module is used to input the video to be identified into a video feature extraction network in a pre-trained object behavior recognition model to obtain global temporal features and global spatial features of the video to be identified, as features to be processed; wherein, the object behavior recognition model further includes a first feature extraction network and a first large language model; the pre-trained object behavior recognition model is trained based on sample videos and sample labels indicating sample behaviors of sample objects in the sample videos; a second feature extraction module is used to use the first feature extraction network to extract image features of the image region occupied by the object to be identified in the video to be identified, as the appearance features of the object to be identified; a first fusion module is used to fuse the appearance features of the object to be identified and the features to be processed to obtain object morphology adaptation features; a recognition result acquisition module is used to perform deep feature extraction on the features to be identified based on the first large language model to obtain the target behavior recognition result of the object to be identified; wherein, the features to be identified are obtained based on the object morphology adaptation features.

[0009] In a fourth aspect of the present invention, a model training apparatus is provided, the apparatus comprising:

[0010] The second acquisition module is used to acquire sample videos and sample labels indicating the sample behavior of sample objects in the sample videos; the third feature extraction module is used to input the sample videos into the video feature extraction network in the object behavior recognition model of the initial structure to obtain the global temporal features and global spatial features of the sample videos as the first sample features; wherein, the object behavior recognition model further includes the first feature extraction network and the first large language model; the fourth feature extraction module is used to use the first feature extraction network to extract the image features of the image region occupied by the sample objects in the sample videos as the appearance features of the sample objects; the second fusion module is used to perform feature fusion on the appearance features of the sample objects and the first sample features to obtain sample morphology adaptation features; the prediction result acquisition module is used to perform deep feature extraction on the second sample features based on the first large language model to obtain the predicted behavior recognition result of the sample objects; wherein, the second sample features are obtained based on the sample morphology adaptation features; the parameter adjustment module is used to adjust the model parameters of the object behavior recognition model of the initial structure based on the difference between the predicted behavior recognition result and the sample labels until the model converges to obtain the trained object behavior recognition model.

[0011] In a fifth aspect of the present invention, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the object behavior recognition method described in any of the first aspects above, or the model training method described in any of the second aspects above, when executing the program stored in the memory.

[0012] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the object behavior recognition method described in any of the first aspects above, or the model training method described in any of the second aspects above.

[0013] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the object behavior recognition method described in any of the first aspects above, or the model training method described in any of the second aspects above.

[0014] The object behavior recognition method provided in this invention uses global temporal and spatial features extracted from the video features of the video to be recognized, representing the global features of the video. The appearance features of the object to be recognized are the image features of the image region occupied by the object in the video, representing the local features of the video. The appearance features and the features to be processed are fused to obtain object morphology adaptation features. Subsequently, the target behavior recognition result is obtained based on these object morphology adaptation features. Compared to using only the features to be processed to obtain the target behavior recognition result, this method can guide the object behavior recognition model to focus more on the object itself, improving the accuracy of the obtained target behavior recognition result, without losing global features (i.e., without losing information about the video itself). Furthermore, when there are stylistic differences between the object to be recognized and the sample object, obtaining the target behavior recognition result based on the object morphology adaptation features can reduce the interference caused by stylistic differences and improve the accuracy of the obtained object behavior recognition result. Thus, the object behavior recognition model can recognize the behavior of objects with various stylistics, indicating high versatility. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0016] Figure 1 This is a first flowchart of an object behavior recognition method provided in an embodiment of the present invention;

[0017] Figure 2 This is a second flowchart of the object behavior recognition method provided in an embodiment of the present invention;

[0018] Figure 3 This is a third flowchart of the object behavior recognition method provided in the embodiments of the present invention;

[0019] Figure 4 A schematic diagram of an object behavior recognition method provided in an embodiment of the present invention;

[0020] Figure 5 A flowchart of a model training method provided in an embodiment of the present invention;

[0021] Figure 6 A structural diagram of an object behavior recognition device provided in an embodiment of the present invention;

[0022] Figure 7 A structural diagram of a model training device provided in an embodiment of the present invention;

[0023] Figure 8 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0025] In the field of video processing technology, an object behavior recognition model with an initial structure can be trained using sample videos containing sample objects and sample labels indicating the behavior of the sample objects until the model converges, resulting in a trained object behavior recognition model. Then, the trained object behavior recognition model is used to identify the behavior of the object to be identified in the video. In real-world scenarios, there may be situations where the style of the sample objects in the sample video differs significantly from the style of the object to be identified. Style can indicate how the object is displayed in the video.

[0026] For example, a video can be an anime, and the subject can be a character from the anime (which can be called an anime character). The art style of the anime can indicate the anime creator's drawing style for the character. If all anime characters are human; the body proportions of the humans in anime A are close to the actual body proportions of humans in real-world scenes (which can be called normal proportions); while the heads of the humans in anime B are larger than the torsos (which can be called chibi style), then anime A and anime B have different art styles.

[0027] For example, the video could be a puppet show, and the object could be the puppet in the puppet show; the style of the puppet show can indicate the sculptor's carving style for the puppet. If the proportions of the puppet's facial features in puppet show C are close to the proportions of human facial features in a real scene, while the eyes of the puppet in puppet show D occupy a larger proportion of the puppet's face area, while the nose and mouth occupy a smaller proportion, then the styles of puppet show C and puppet show D are also different.

[0028] For example, the video could be a TV series starring live actors, and the subject could be a live character in the series; the style of the TV series could indicate the clothing of the live characters. For instance, the live characters in TV series E are wearing period costumes; the live characters in TV series F are wearing modern clothes. Accordingly, the styles of TV series E and TV series F are also different.

[0029] In real-world scenarios, videos can encompass various types, including anime, puppet shows, and TV dramas. The stylistic differences between objects in different types of videos are greater than those in videos of the same type. Furthermore, the style of the same object within a single video can change; for example, character 1 in anime A might appear in a normal form initially but later transform into a chibi (Q-version) version. In other words, the stylistic diversity of objects in real-world scenarios is considerable.

[0030] Since object behavior recognition models are trained on sample videos, the accuracy of the recognition results is high when the style of the object to be recognized is similar to that of the sample object. However, if the style of the object to be recognized differs significantly from that of the sample character—for example, if the sample object is a live-action character and the object to be recognized is an anime character—the accuracy of the object behavior recognition results obtained from the model may be lower. In other words, the general applicability of existing object behavior recognition models is limited.

[0031] To address the aforementioned problems, this invention provides an object behavior recognition method applied to electronic devices. The electronic device can be a server, such as a backend server of a video platform. Based on the object behavior recognition method provided by this invention, the server can more accurately identify the behavior of characters in a video, and subsequently perform video review based on the identified behavior. Alternatively, the electronic device can also be a client, such as a mobile device used by a user who publishes a video. The client, based on the object behavior recognition method provided by this invention, can more accurately identify the behavior of characters in the video to be published, and then add tags to the video based on the identified behavior. Subsequently, other users can browse the videos published by that user based on the tags. The actual scenarios are not limited to these examples, and this invention does not limit them.

[0032] See Figure 1 , Figure 1This is a first flowchart of an object behavior recognition method provided in an embodiment of the present invention. The method may include the following steps:

[0033] S101: Obtain the video to be identified.

[0034] S102: Input the video to be identified into the video feature extraction network of the pre-trained object behavior recognition model to obtain the global temporal features and global spatial features of the video to be identified, which are used as features to be processed.

[0035] The object behavior recognition model also includes a first feature extraction network and a first large language model; the pre-trained object behavior recognition model is trained based on sample videos and sample labels that indicate the sample behavior of sample objects in the sample videos.

[0036] S103: Using the first feature extraction network, extract the image features of the image region occupied by the object to be identified in the video to be identified, and use it as the appearance features of the object to be identified.

[0037] S104: Perform feature fusion on the appearance features and features to be processed of the object to be identified to obtain object morphology adaptation features.

[0038] S105: Based on the first major language model, perform deep feature extraction on the features to be identified to obtain the target behavior recognition result of the object to be identified.

[0039] Among them, the features to be identified are obtained based on the object morphology adaptation features.

[0040] Based on the above processing, the features to be processed are the global temporal and global spatial features obtained from video feature extraction of the video to be recognized, which can represent the global features of the video to be recognized; while the appearance features of the object to be recognized are the image features of the image region occupied by the object to be recognized in the video to be recognized, which can represent the local features of the video to be recognized. The appearance features of the object to be recognized and the features to be processed are fused to obtain object morphology adaptation features; subsequently, the target behavior recognition result is obtained based on the object morphology adaptation features. Compared to using only the features to be processed to obtain the target behavior recognition result, this approach can guide the object behavior recognition model to focus more on the object to be recognized itself based on local features without losing global features, i.e., without losing information about the video itself, thus improving the accuracy of the obtained target behavior recognition result. Furthermore, when there are differences in the style between the object to be recognized and the sample object, obtaining the target behavior recognition result based on the object morphology adaptation features can reduce the interference caused by style differences and improve the accuracy of the obtained object behavior recognition result. Thus, the object behavior recognition model can recognize the behavior of objects with various style requirements, meaning that the object behavior recognition model has high versatility.

[0041] Regarding step S101, in one approach, the video acquired by the electronic device can be directly used as the video to be identified. For example, a user can edit a video already published online, and the electronic device can acquire the user's editing results; a user can also manually draw each video frame, and the electronic device can acquire the video composed of the user's drawn video frames. Alternatively, the electronic device can also acquire the video to be released by the distributor of the entertainment work. The video acquired by the electronic device can be in MP4 (Moving Picture Experts Group-4 Part 14) format, FLV (Flash Video) format, etc., and this invention does not limit this. In this way, the probability of video content loss can be reduced, and the accuracy of the subsequent target behavior recognition results can be improved.

[0042] In another approach, the electronic device can preprocess the acquired video and use the preprocessed result as the video to be identified. Preprocessing includes at least one of the following: temporal sampling and spatial processing.

[0043] Specifically, electronic devices can first perform video decoding on the acquired video to convert it into a sequence of consecutive video frames (which can be called raw video frames). For example, electronic devices can perform video decoding based on FFmpeg (FastForward Moving Picture Experts Group) technology.

[0044] Temporal sampling involves sampling the sequence. For example, an electronic device can sample multiple video frames from the sequence at a preset sampling interval; alternatively, it can sample a specified number of video frames. The video composed of the sampled video frames is then used as the video to be identified. Both the preset sampling interval and the specified number can be set based on the needs of the actual scenario. For example, the preset sampling interval can be 20 frames, and the specified number can be 32, 64, 128, etc. This invention does not limit these settings.

[0045] Spatial processing involves compressing video frames within a sequence. For example, an electronic device might compress video frames to a specified size, which can be denoted as (H, W), where H represents the height of the compressed image and W represents its width. In practical scenarios, one of the specified dimensions H and W is smaller than the original video frame, while the other is no larger. For instance, the original video frame size could be (1280, 1280), and the specified size could be (224, 224), which can be represented as 224×224.

[0046] In practical scenarios, electronic devices can first perform temporal sampling on the decoded sequence, and then perform spatial processing on the sampled video frames; the video composed of the spatially processed video frames is used as the video to be identified. The video to be identified can be represented in the form of a video frame tensor, which can be denoted as (N, C, H, W); N represents the aforementioned number, and C represents the number of color channels in each video frame. For example, when the video frame is in RGB (Red, Green, Blue) format, C is 3.

[0047] In this way, by preprocessing the video acquired by the electronic device to obtain the video to be recognized, the computational load of subsequent processing can be reduced and the efficiency of object behavior recognition can be improved.

[0048] Regarding step S102, a pre-trained object behavior recognition model can be deployed in the electronic device, allowing the device to use this locally deployed model for subsequent processing. The training method for the object behavior recognition model is detailed in subsequent embodiments and will not be repeated here.

[0049] The object behavior recognition model includes a video feature extraction network for extracting video features from the video to be recognized, obtaining global temporal and global spatial features as features to be processed. Global temporal features represent the optical flow information between every two adjacent video frames in the video to be recognized, while global spatial features represent the image features of each video frame in the video to be recognized. For example, the video feature extraction network can be C3D or TimeSFormer, built based on Transformer. Alternatively, the video feature extraction network can include a temporal feature extraction network for extracting global temporal features from the video to be recognized, and a spatial feature extraction network for extracting global spatial features from the video to be recognized. The temporal feature extraction network can be an RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), etc.; the spatial feature extraction network can be a CNN (Convolutional Neural Network), etc., and this invention does not limit this to these methods.

[0050] Taking TimeSFormer as an example of a video feature extraction network, the electronic device can acquire the input data of the Transformer encoder in TimeSFormer in a similar manner to acquiring the input data of a VisionTransformer (VIT). Specifically, for each video frame included in the video to be recognized, the video frame is divided into non-overlapping patches. The position of an image patch in the video to be recognized includes: the position of the image patch in its own video frame (i.e., the spatial position of the image patch in the video to be recognized), and the position of the video frame to which the image patch belongs in the video to be recognized (i.e., the temporal position of the image patch in the video to be recognized). Correspondingly, an image patch can also be called a spatiotemporal block. Furthermore, the electronic device can generate a spatiotemporal position code indicating the temporal and spatial positions of the image patch in the video to be recognized. Then, the spatiotemporal position code corresponding to the image patch is added to the image patch. For each image patch, a linear projection method is used to map the pixel matrix of that image patch into a vector (which can be called the flattened vector corresponding to that image patch). The flattened vector corresponding to that image patch and the spatiotemporal location code are then concatenated to obtain a feature concatenation result that retains the original location information of that image patch in the video to be recognized. The feature concatenation results corresponding to each image patch are the input data for the Transformer encoder in TimeSFormer.

[0051] The Transformer encoder processes the feature stitching results corresponding to each image patch based on a multi-head self-attention mechanism. It can learn the global spatiotemporal dependencies between each image patch and then extract the features of each image patch, including the aforementioned global spatiotemporal dependencies, which are the features to be processed in the video to be recognized.

[0052] In one approach, to reduce the computational load of subsequent processing, the electronic device can also perform pooling on the features output by the Transformer encoder, using the pooling result as the features to be processed in the video to be recognized (also known as compact features), which can be denoted as F. video Pooling can be performed using methods such as max pooling or average pooling; F video It can also be called video spatiotemporal feature vector.

[0053] Regarding step S103, the video to be identified includes at least one object to be identified. The object to be identified can be any object in the video, or it can be a specific object within the video. For example, it could be the object with the highest resolution in the video, or the object whose image area occupies the largest proportion in each video frame.

[0054] The object behavior recognition model also includes a first feature extraction network, used to extract image features of the image region (hereinafter referred to as the object region) occupied by the object to be recognized in the video to be recognized. For example, for each video frame in the video to be recognized, the object region is the image region within the bounding rectangle of the object to be recognized. Since the object to be recognized is usually displayed in the video frame, it usually displays appearance information such as hair color, facial features, body proportions, and clothing. Therefore, extracting the image features of the object region, that is, extracting the appearance features of the object to be recognized in the object region (hereinafter referred to as the appearance features to be recognized), can be denoted as F. form For example, the first feature extraction network may include an image region detection layer and a region feature extraction layer. For each video frame in the video to be identified, the image region detection layer is used to detect the image region occupied by the object to be identified in that video frame; then, the region feature extraction layer is used to extract the image features of the detected image region.

[0055] Regarding steps S104 and S105, after acquiring the appearance features to be identified and the features to be processed, the object behavior recognition model can perform feature fusion on the appearance features to be identified and the features to be processed. For example, an electronic device can perform feature splicing on the appearance features to be identified and the features to be processed according to feature channels to obtain object shape adaptation features.

[0056] The first major language model includes multiple feature extraction layers for feature extraction. Using multiple network layers in the first major language model for feature extraction can be called deep feature extraction. For example, if the first major language model includes 30 feature extraction layers, using the first 25 of these layers to extract features from the target features constitutes deep feature extraction.

[0057] Since the feature dimensions of object morphological adaptation features in real-world scenarios may not be supported by the first language model, electronic devices can map the object morphological adaptation features according to the dimensions supported by the first language model in order to subsequently use the first language model to obtain target behavior recognition results. This yields the features to be recognized that can be processed by the first language model. Then, deep feature extraction is performed based on the features to be recognized using the first language model to obtain the target behavior recognition result, i.e., the behavior of the object to be recognized in the video. For example, the target behavior recognition result can include "dancing," "eating," "sleeping," "fighting," "hugging," and "arguing."

[0058] In some embodiments, the first feature extraction network is an object detection network; the appearance features of the object to be identified are obtained by deep feature extraction of the video to be identified through the object detection network.

[0059] In this embodiment of the invention, the object detection network can be based on an object detection model, such as the network used for feature extraction in the YOLO (You Only Look Once) model; or it can include the network used for feature extraction in other object detection models.

[0060] An object detection model can include multiple feature extraction layers for feature extraction, and an object detection network can include some of these feature extraction layers. For example, the YOLO model includes 20 feature extraction layers. If the first 18 feature extraction layers of the YOLO model are used to extract features from the video to be identified (i.e., deep feature extraction), then the object detection network can include the first 18 feature extraction layers of the YOLO model. Correspondingly, the object detection network processes the video to be identified, i.e., performs deep feature extraction, to obtain the appearance features to be identified.

[0061] Based on the above processing, electronic devices can directly acquire the appearance features to be identified based on the target detection network, without first determining the image area occupied by the object to be identified in the video and then extracting the image features of the determined image area. This simplifies the model structure, improves the efficiency of acquiring the appearance features to be identified, and thus improves the efficiency of the target behavior recognition results.

[0062] In some embodiments, the aforementioned step S104 may include the following steps:

[0063] Step A1: Divide the appearance features of the object to be identified to obtain the target scaling factor and the target bias factor.

[0064] Step A2: Scale the feature to be processed using the target scaling factor to obtain the target scaling result.

[0065] Step A3: Calculate the sum of the target bias coefficient and the target scaling result to obtain the object shape adaptation feature.

[0066] In this embodiment of the invention, the appearance feature to be identified is divided into two coefficients. These two coefficients are used to adjust the feature to be processed, thereby achieving the fusion of the appearance feature to be identified and the feature to be processed. Therefore, this embodiment of the invention does not limit the specific method of dividing the appearance feature to be identified to obtain the two coefficients. For example, the appearance feature to be identified can be divided into two matrices of the same size. For each matrix obtained, the matrix is ​​mapped to the feature dimension of the feature to be processed (hereinafter referred to as the target dimension), resulting in a one-dimensional vector of the target dimension. Then, the two one-dimensional vectors obtained are used as the target scaling coefficient and the target bias coefficient, respectively. The matrices obtained from the division can be denoted as matrix A and matrix B; the one-dimensional vector obtained based on matrix A can be denoted as vector a, and the one-dimensional vector obtained based on matrix B can be denoted as vector b. Vector a can be used as the target scaling coefficient, and vector b can be used as the target bias coefficient; or, vector b can be used as the target scaling coefficient, and vector a can be used as the target bias coefficient, both of which are reasonable.

[0067] Furthermore, the features to be processed are dynamically adjusted using the target scaling factor and the target bias factor. The processing procedure of the feature fusion layer can be expressed as the following formula (1):

[0068] F adapted_form =γ·F video +β; (1)

[0069] Among them, F adapted_form Represents the object's form adaptation feature (which can be called the dynamic character form adaptation feature); γ represents the target scaling factor, β represents the target bias factor; F video This indicates the feature to be processed.

[0070] Based on the above processing, by dividing the appearance feature to be identified into two coefficients, and then using these two coefficients to adjust the feature to be processed, feature fusion is achieved between the feature to be processed and the appearance feature to be identified. That is, the appearance feature to be identified guides the adjustment method of the feature to be processed. Subsequently, based on the obtained object morphology adaptation features, the target behavior recognition result can be obtained, which can guide the object behavior recognition model to pay more attention to the information of the object itself, thereby improving the accuracy of the obtained target behavior recognition result.

[0071] In some embodiments, the object behavior recognition model further includes a first mapping network.

[0072] Step A1 mentioned above may include the following steps:

[0073] Step B1: Using the first mapping network, the appearance features of the object to be identified are mapped to a specified dimension to obtain the target-fitting features.

[0074] The specified dimension is twice the feature dimension of the feature to be processed.

[0075] Step B2: Divide the target adaptation features into two vectors, which will be used as the target scaling factor and the target bias factor, respectively.

[0076] In this process, the feature dimension of each vector obtained by partitioning is the same as the feature dimension of the feature to be processed.

[0077] In this embodiment of the invention, the output dimension of the first mapping network is a specified dimension. By mapping the appearance features to be identified through the first mapping network, the appearance features to be identified can be mapped to the specified dimension to obtain the target fitting features. The feature dimension of the feature to be processed (i.e., the aforementioned target dimension) can be denoted as C, and correspondingly, the specified dimension can be denoted as 2×C. The target fitting features are one-dimensional vectors containing 2×C elements.

[0078] In some embodiments, the first mapping network includes a transformation network and a first multilayer perceptron.

[0079] Step B1 may include the following steps:

[0080] Step C1: Using a transformation network, a self-attention mechanism is used to extract the appearance features of the object to be identified, thereby obtaining the object's morphological features.

[0081] Step C2: Use the first multilayer perceptron to perform feature adaptation processing on the object's morphological features to obtain features of a specified dimension, which are then used as target adaptation features.

[0082] In this embodiment of the invention, the transformation network is a network built based on a self-attention mechanism, such as a Transformer encoder. The appearance features to be recognized are input into the transformation network, which uses a self-attention mechanism to extract features from the appearance features to be recognized. This can further remove redundant features (such as features of the background region in the object region) from the appearance features to be recognized, resulting in more accurate appearance features to be recognized, which are then used as the morphological features to be recognized.

[0083] Then, a first multilayer perceptron is used to perform feature adaptation processing on the morphological features to be identified. Specifically, the first multilayer perceptron may include an input layer, a hidden layer, and an output layer. The input layer receives the morphological features to be identified as input data to the first multilayer perceptron. The hidden layer may include at least one fully connected layer, and each fully connected layer may be followed by a non-linear activation function, such as a ReLU activation function. The fully connected layer included in the hidden layer is used to extract the non-linear features from the morphological features to be identified. The output layer includes a fully connected layer, and this fully connected layer does not include an activation function; the number of output channels of this fully connected layer is 2×C. The output layer is used to process the non-linear features extracted by the hidden layer. After the first multilayer perceptron performs a complete forward operation, the output layer can output a one-dimensional vector containing 2×C elements, that is, obtain the target adaptation feature of the specified dimension.

[0084] Then, the target adaptation feature is divided into two equal-length parts, resulting in two one-dimensional vectors. These two one-dimensional vectors can be used as the target scaling factor and the target bias factor, respectively. For example, the vector consisting of the first C elements of the target adaptation feature can be used as the target scaling factor, and the vector consisting of the remaining C elements can be used as the target bias factor; alternatively, the vector consisting of the first C elements of the target adaptation feature can be used as the target bias factor, and the vector consisting of the remaining C elements can be used as the target scaling factor; alternatively, the vector consisting of the odd-numbered elements of the target adaptation feature can be used as the target scaling factor, and the vector consisting of the even-numbered elements can be used as the target bias factor; alternatively, the vector consisting of the odd-numbered elements of the target adaptation feature can be used as the target bias factor, and the vector consisting of the even-numbered elements can be used as the target scaling factor. All of these are reasonable.

[0085] Based on the above processing, while making full use of the appearance features to be identified, two coefficients can be obtained through a single mapping. This simplifies the model structure of the object behavior recognition model, reduces the computational load of obtaining the two coefficients, improves the computational efficiency of calculating the two coefficients, and thus improves the efficiency of the object behavior recognition model.

[0086] This method, which acquires target adaptation features of a specified dimension based on a transformation network and a first-layer perceptron, divides these features into target scaling factors and target bias factors, and then adjusts the features to be processed based on these factors to obtain object shape adaptation features, can be called FiLM (Feature-wise Linear Modulation). It employs a gated attention mechanism or conditional normalization method to dynamically adjust the features to be recognized based on the appearance features F. form Adjust the feature F to be processed video An adaptive feature fusion processing method.

[0087] In some embodiments, the object behavior recognition model further includes a performance technique classification network and a second multilayer perceptron.

[0088] exist Figure 1 Based on this, see Figure 2 Prior to step S105, the method may further include the following steps:

[0089] S106: Input the features to be processed into the performance technique classification network to obtain the technique category representing the video content performance technique of the video to be identified.

[0090] S107: Concatenate the identifiers of the feature to be processed and the corresponding method category to obtain the target concatenation result.

[0091] S108: The target splicing result is processed by the second multilayer perceptron to obtain the performance technique adaptation feature.

[0092] S109: Perform feature fusion on the object morphology adaptation features and the expression technique adaptation features to obtain the features to be identified.

[0093] In this embodiment of the invention, a representation method classification network can also be used to obtain the method category representing the video content representation method of the video to be identified. The video content representation method of the video to be identified can indicate the way the behavior of the object to be identified is displayed in the video to be identified.

[0094] For example, when the video to be identified is an animation, the behavior of the animation character can be "running"; the animation creator can draw multiple parallel lines behind the animation character to indicate that the animation character is "running". The above parallel lines can be called speed lines, and this representation technique can be called "speed line appearance".

[0095] For example, an anime character's behavior could be "receiving a message"; the anime creator could draw multiple lines on the anime character's face to indicate that the character's expression is "distorted" to suggest that the character has received a shocking message. This technique can be called "extremely distorted expression".

[0096] For example, an anime character's action could be "releasing magic"; the anime creator could draw a ball of light in the direction the anime character is facing to represent the anime character "releasing magic," and this technique could be called a "shockwave" or "magic release."

[0097] For example, an anime character's behavior can be described as "going berserk"; the anime creator can draw flames next to the anime character's body area to indicate that the anime character can be in a state of "anger" or "energy burst", and this expression can be called "the character is on fire".

[0098] For example, when the video to be identified is a TV series, the behavior of the real-life character can be "transformation"; the real-life character in the TV series can wear clothing 1 when he appears; after the real-life character's body area is covered by a ball of light, the real-life character can wear clothing 2 when he reappears. This performance technique can be called "transformation".

[0099] The performance technique classification network is used to determine the performance technique category (hereinafter referred to as the target technique category) of the video content that matches the feature to be processed from various preset technique categories. The performance technique classification network can be a VIT model, or it can be other classification models. This invention does not limit this.

[0100] The preset technique categories can be set by technicians based on their work experience. For example, they can include the aforementioned "speed line appears", "extreme facial distortion", "shock wave", "magic release", "character's whole body is on fire", "transformation", or other technique categories that may appear in actual scenarios. This invention does not limit these categories.

[0101] The aforementioned performance techniques can also be called exaggerated performance techniques; the aforementioned performance technique classification network can be called an exaggerated pattern recognizer. The performance technique classification network determines the technique category corresponding to the video to be recognized, and can obtain the identifier corresponding to the technique category (which can be called exaggerated information encoding), denoted as F. exaggeration_info .

[0102] The identifiers of the techniques and their corresponding categories are concatenated, for example, by feature channels, to obtain the target concatenation result. Then, a second multilayer perceptron performs feature adaptation processing on the target concatenation result to obtain the technique adaptation features, denoted as F, which indicate the technique categories of the features to be processed and the video to be identified. adapted_exaggeration Thus, the performance techniques can be converted into semantic information that is beneficial to the understanding of the object behavior recognition model. Correspondingly, the process of obtaining the performance technique adaptation features based on the identifier corresponding to the technique category can be called the semantic enhancement process, which can be expressed as the following formula (2):

[0103] F adapted_exaggeration =f(F video ,F exaggeration_info (2)

[0104] Since object morphology adaptation features are obtained by adapting the image features of the image region occupied by the object to be identified, the object morphology features are adapted visual features; while performance technique adaptation features represent the semantic information obtained by semantic enhancement, the performance technique adaptation features are adapted semantic features. Therefore, feature fusion of object morphology adaptation features and performance technique adaptation features can be called visual-linguistic multimodal feature fusion. For example, object morphology adaptation features and performance technique adaptation features can be concatenated, and then the concatenated feature results can be mapped according to the dimensions supported by the first major language model, and the mapped results can be used as the features to be identified.

[0105] In some embodiments, the object behavior recognition model further includes a second mapping network and a second major language model.

[0106] The aforementioned step S109 may include the following steps:

[0107] Step D1: Perform feature fusion on the object morphology adaptation features and the expression technique adaptation features to obtain the first visual fusion features.

[0108] Step D2: Using the second mapping network, the first visual fusion features are mapped according to the dimensions supported by the second largest language model to obtain the second visual fusion features.

[0109] Step D3: Perform cross-modal alignment on the second visual fusion features to obtain the target prompt information supported by the second largest language model.

[0110] Step D4: Use the second language model to perform deep feature extraction on the target prompt information to obtain the features to be identified.

[0111] In this embodiment of the invention, the electronic device performs feature fusion on object shape adaptation features and performance technique adaptation features (hereinafter collectively referred to as adaptation features). For example, after concatenating the two adaptation features, a Transformer is used to extract features from the concatenated result, and the extracted feature result is used as the first visual fusion feature; or, the electronic device can map any adaptation feature to a specified dimension, then divide the mapping result into two coefficients, where the feature dimensions of both coefficients are the target dimension, and then use the divided coefficients to adjust another adaptation feature, using the adjustment result as the first visual fusion feature. Both of these are reasonable. The first visual fusion feature can be denoted as F. input_visual The above process is called the feature preparation process.

[0112] Then, a second mapping network is used for feature alignment to adjust F. input_visual Mapping to the dimension supported by the second largest language model, the mapping result can be denoted as F'. input_visualIn this way, the second language model can be used for further processing based on the mapping results. The second mapping unit can be a linear projection layer.

[0113] Furthermore, based on Q-former (cross-modal alignment technology), vision-language fusion can be performed. Specifically, electronic devices can be configured with learnable query vectors Q... learnable Before performing cross-modal alignment, Q learnable The parameters in Q can be preset, such as all being 0. A cross-attention mechanism is used to apply attention to Q. learnable and F' input_visual Perform feature extraction and adjust Q using the extracted features. learnable The parameters in the adjusted Q learnable Q, which has already undergone one learning session learnable Then, continuing based on the Q that has already been learned once... learnable and F' input_visual Perform feature extraction until a stopping condition is met, such as Q. learnable After a predetermined number of learning iterations, the latest obtained Q... learnable As a result of cross-modal alignment, it can be denoted as F qureied_visual F qureied_visual The visual features that the second largest language model can process (i.e., target cue information) can also be called soft cues. The above visual-language fusion process can be expressed as the following formula (3):

[0114] F qureied_visual =Qformer(Q learnable , F' input_visual (3)

[0115] Furthermore, a second large language model is used to extract deep features from the target prompt information. This second large language model can be the LLM part of a pre-trained MLLM (Multimodal Large Language Model) that has already learned pre-trained knowledge, specifically MiniCPMV-2.6 (Miniature Contrastive Prompting for MultimodalVision 2.6). During pre-training, MLLM can learn scene information (such as battlefield, school, park, etc.) and character dialogue. Using the pre-trained knowledge and reasoning ability of the second large language model, semantic features (also known as hidden state representations) containing semantic information are extracted from the target prompt information that integrates visual and semantic information. This can be denoted as H. llmUsing a second-largest language model to obtain the features to be identified can also be described as deeply fusing adapted visual features with the semantic understanding capabilities of MLLM.

[0116] Based on the above processing, the object behavior recognition model fuses adapted visual features and adapted semantic features to obtain the features to be recognized. Subsequently, the target behavior recognition result is obtained based on these features; that is, semantic features are used to assist in the behavior recognition process. Compared to using only visual features to recognize the behavior of the object, using the behavior indicated by semantic features allows the object behavior recognition model to interpret visual features more deeply, resulting in more accurate target behavior recognition results.

[0117] For example, visual features may only determine that "object 1 is moving quickly and object 2 is falling down", but it is difficult to distinguish whether object 1 and object 2 are "fighting" or "playing". However, by combining semantic features, such as the semantic feature corresponding to the expression method "the character is on fire", which represents that the object is "going berserk", the object behavior recognition model can obtain the target behavior recognition result indicating that object 1 and object 2 are "fighting".

[0118] In some embodiments, a technician may manually set the correspondence between video content and technique categories. The electronic device can then find the technique category corresponding to the video content of the video to be identified from this correspondence, and use it as the target technique category. For example, the technician can set the maximum brightness difference between adjacent frames in the video to correspond to technique category 1 if it falls within interval 1, and technique category 2 if it falls within interval 2. The electronic device can calculate the maximum brightness difference between adjacent frames in the video to be identified and use the technique category corresponding to the interval to which the maximum value belongs as the target technique category.

[0119] In some embodiments, Figure 2 Based on this, see Figure 3 The aforementioned step S105 may include the following steps:

[0120] S1051: Utilize the deep feature extraction layer in the first language model to perform deep feature extraction on the features to be identified, and obtain the features to be utilized.

[0121] S1052: Calculate the matching probability between the feature to be used and each behavior label using the label classification layer in the first language model.

[0122] S1053: Among all behavior labels, the behavior indicated by the behavior label with a matching probability greater than the feature to be utilized is taken as the first target behavior recognition result of the object to be identified.

[0123] In this embodiment of the invention, the deep feature extraction layer, i.e., the feature extraction layer of the first large language model in the aforementioned embodiments, is used to perform deep feature extraction on the features to be identified. For example, based on the aforementioned example, the deep feature extraction layer can be the 25th feature extraction layer in the first large language model. The features output by the 25th feature extraction layer are the features to be utilized by the deep feature extraction.

[0124] Electronic devices can directly use the deep feature extraction layer in the first major language model to perform deep feature extraction on the features to be recognized, obtaining the features to be utilized; or, to reduce computation and improve the efficiency of object behavior recognition, the features to be recognized can be pooled first to obtain pooled features, denoted as H. pooled_llm Deep feature extraction is performed on the pooling features to obtain the features to be utilized.

[0125] The label classification layer in the first major language model may include at least one fully connected layer, and each fully connected layer may be followed by an activation function, such as a sigmoid activation function, to calculate the matching probability between the feature to be utilized and a preset set of M behavioral labels. A behavioral label indicates a class of behavior, such as the aforementioned "fighting," "transformation," and "contemplation." Behavioral labels can be set by technical personnel based on their work experience. This invention does not limit the number of behavioral labels, nor does it restrict the specific behavior indicated by the labels.

[0126] The label classification layer can output a probability vector, denoted as P = {p1, p2, p3, ..., pm}; px represents the matching probability between the feature to be utilized and the preset x-th behavior label.

[0127] Furthermore, the electronic device can identify the behavior labels from the various behavior labels whose matching probability with the feature to be utilized is greater than a preset probability threshold, and use these as target behavior labels. The behavior indicated by the target behavior label is taken as the first target behavior recognition result of the object to be identified. The preset probability threshold can be set by technicians based on their work experience, such as 0.8. In actual scenarios, the matching probability of each behavior label with the feature to be utilized may not be greater than the preset probability threshold, or there may be multiple behavior labels whose matching probability with the feature to be utilized may be greater than the preset probability threshold.

[0128] The preset behavior labels can be called closed sets. Based on the embodiments of the present invention, the object behavior recognition model can determine the behavior of the object to be identified from the behaviors already included in the closed set. This process can directly serve the business needs in real-world scenarios. For example, the closed set may include "fighting." When a video platform conducts video review, it needs to remove videos containing the object "fighting." The reviewers of the video platform can directly remove videos whose target behavior recognition results identified by the object behavior recognition model include "fighting," thereby improving the efficiency of video review.

[0129] For example, when recommending videos, if a video platform determines that a target user has a high level of interest in videos featuring an object "dancing" based on the user profile of the target user, the video platform can recommend videos featuring "dancing" based on the target behavior recognition results identified by the object behavior recognition model to the target user, thereby improving the effectiveness of recommending videos to the target user.

[0130] In some embodiments, after the aforementioned step S1052, the method may further include the following steps: using the text output layer in the first large language model to decode the features to be used, and obtain a second target behavior recognition result indicating the behavior of the object to be identified.

[0131] In this embodiment of the invention, since the behavioral labels included in the aforementioned closed set are limited and difficult to cover all the behaviors of objects that may appear in the actual scene, in order to reduce the probability that the behavior of the object to be identified is not detected, the electronic device can also input descriptive text into the object behavior recognition model to instruct the object behavior recognition model to use natural language to describe behaviors not included in the closed set. For example, the descriptive text can be "Please describe the unrecognized object behavior in the video".

[0132] The text output layer in the first language model can act as a decoder to decode the features extracted from the first language model. Electronic devices can convert the aforementioned descriptive text into tokens (text units), and input the tokens and the features to be recognized into the first language model. The deep feature extraction layer in the first language model can perform deep feature extraction on the tokens to obtain token features; the text output layer in the first language model decodes the token features and the features to be recognized, obtaining the output text of the first language model, which serves as the result of the second target behavior recognition. For example, the output text could be "arguing". This process can be called using LLM for open description; the output text can be called the open set.

[0133] In real-world scenarios, the first language model can also output indications that the video to be identified contains behaviors not included in a closed set. For example, the first language model can output "Unrecognized object behaviors in the video include: arguing." The text before "arguing" is the aforementioned indication information.

[0134] In real-world scenarios, the object behavior recognition model can display to the user all the first target behavior recognition results obtained based on the aforementioned closed set, as well as all the second target behavior recognition results obtained based on the open set, to generate the final business requirement behavior label list (i.e., the first target behavior recognition results) and output the detected non-business requirement behavior information (i.e., the second target behavior recognition results).

[0135] In some embodiments, the electronic device inputs the aforementioned token features and the aforementioned target prompt information into the second language model. The aforementioned descriptive text assists the second language model in performing deep feature extraction, which can guide the second language model to extract more accurate features to be identified, thereby improving the accuracy of the subsequent target behavior recognition results.

[0136] See Figure 4 , Figure 4 This is a schematic diagram illustrating a method for object behavior recognition provided in an embodiment of the present invention. In this embodiment, the video to be recognized is an anime video clip, and the object to be recognized is an anime character. This method can be implemented through the collaboration of five modules.

[0137] The video input and preprocessing module is used to perform the aforementioned processing of acquiring the video to be identified. Preprocessing includes at least one of the aforementioned temporal sampling and spatial processing. The preprocessed frame sequence obtained by the video input and preprocessing module is each video frame in the video to be identified in the aforementioned embodiments.

[0138] The video feature extraction module, namely the aforementioned video feature extraction network, is used to extract the global temporal and global spatial features of the video to be identified, thereby obtaining the features F to be processed. video (Right now Figure 4 (Video features in the video).

[0139] The animation feature adaptive module includes: a dynamic character form adaptation submodule and an exaggerated behavior understanding submodule.

[0140] The dynamic character form adaptation submodule, namely the first feature extraction network in the aforementioned embodiment, is used to extract form adaptation features, namely the aforementioned object form adaptation feature F. adapted_form .

[0141] The exaggerated behavior understanding submodule includes the performance technique classification network and the second multilayer perceptron in the aforementioned embodiments, used to obtain exaggerated understanding features, namely the aforementioned performance technique adaptation features F. adapted_exaggeration .

[0142] The Multimodal Fusion Module, including the second mapping network and the second major language model in the aforementioned embodiments, is used to fuse object morphological adaptation features and performance technique adaptation features to obtain fused features F. fused That is, the aforementioned feature H to be identified llm .

[0143] The Behavior Classification Output Module, also known as the first language model in the aforementioned embodiments, is used to perform deep feature extraction on the features to be identified, obtaining the target behavior recognition result. It includes a business requirement behavior classification head (i.e., the label classification layer in the aforementioned embodiments) and a non-business requirement behavior recognition and description module (i.e., the text output layer in the aforementioned embodiments). Business requirement behavior labels represent the first target behavior recognition result in the aforementioned embodiments; non-business requirement behavior labels represent behaviors that do not belong to predefined business labels, i.e., the second target behavior recognition result in the aforementioned embodiments. This process can also be called dual-channel behavior classification and discrimination.

[0144] Based on the above processing, the dynamic character morphology adaptation submodule, through character perception and morphological encoding, enables the model to focus on the essence of behavior and reduce the interference caused by changes in character appearance; this can be understood as purifying the behavioral characteristics of anime characters. The exaggerated behavior understanding submodule, by identifying exaggerated patterns and enhancing semantic understanding, enables the model to correctly interpret the exaggerated expressions unique to anime; this can be understood as enhancing the behavioral characteristics of anime characters. The multimodal fusion module combines video features adapted to anime characteristics with the powerful understanding capabilities of a language model to achieve in-depth analysis and accurate judgment of anime behavior. The behavior classification output module can meet the differentiated needs in real-world scenarios.

[0145] By incorporating an animation characteristic adaptive module that includes a dynamic character form adaptation submodule and an exaggerated behavior understanding submodule, the unique characteristics of animation are no longer treated as noise but rather as key information. The object behavior recognition model actively encodes and understands this key information, enabling it to learn the special visual language of animation and the same or related behaviors of animation characters in different forms. This overcomes the problem of low accuracy in traditional models caused by factors such as character transformations, changes in animation style, and unique animation techniques. Even in complex animation scenarios with varied character forms (such as "chi-version," "normal state," and "transformation"), diverse art styles, and exaggerated techniques, it can still achieve highly accurate target behavior recognition results. This improves the accuracy and robustness of the object behavior recognition model.

[0146] Furthermore, by deeply integrating the adapted visual features with MLLM and leveraging MLLM's pre-trained knowledge and powerful contextual reasoning capabilities, the object behavior recognition model can infer the intent, emotion, and causal relationship behind the object's behavior by combining the context. This enables the object behavior recognition model to leap from recognizing behavior to understanding behavior, providing a deep semantic foundation for more advanced intelligent applications.

[0147] Furthermore, through a dual-channel output mechanism combining closed and open sets, it can accurately classify predefined behavioral tags for business purposes while also discovering and describing unknown and novel behaviors. This allows the object behavior recognition model to not only directly serve the needs of existing businesses such as video review and video recommendation, but also continuously mine new content in videos and discover new hot topics based on open sets, driving business innovation from a data perspective. The related ideas and module designs can provide a reference for the analysis of other stylized content, demonstrating high flexibility and scalability.

[0148] In some embodiments, the object behavior recognition model can also extract features from the audio of the video to be recognized, including the speech of the object to be recognized, background music, etc., to obtain audio features of the video to be recognized. Then, cross-modal alignment is performed on both the audio features and the second visual fusion features, and feature fusion is performed on the cross-modal alignment results. The fusion result is used as target cue information. Subsequently, target behavior recognition results are obtained based on the target cue information to further improve the accuracy of the obtained target behavior recognition results.

[0149] In some embodiments, the model structure of the object behavior recognition model can be optimized based on model lightweighting, knowledge distillation, and related technologies of streaming video processing, so as to improve the computational efficiency of the object behavior recognition model and meet the real-time behavior recognition needs in scenarios such as live streaming and real-time interaction.

[0150] In some embodiments, for a series of works, including animations, movies, games, etc. under the same IP (Intellectual Property Rights), the behavior of the same character in different works can also be analyzed to provide data support for IP value assessment and derivative development.

[0151] In some embodiments, the video to be identified can be edited based on the target behavior recognition results to automatically generate marketing materials such as trailers, short videos, and GIFs, thereby reducing the production cost of marketing materials and achieving efficient and large-scale content distribution.

[0152] This invention also provides a model training method, which is also applied to electronic devices. To distinguish it from the aforementioned electronic device for object behavior recognition, the electronic device used to execute the model training method will be hereinafter referred to as a training device. The training device and the aforementioned electronic device for object behavior recognition can be the same device, or they can be different devices; this invention does not limit this. When the training device and the aforementioned electronic device for object behavior recognition are different devices, after the training device obtains a trained object behavior recognition model by executing the model training method provided by this invention, it can send the trained object behavior recognition model to the aforementioned electronic device, so that the aforementioned electronic device, after deploying the received object behavior recognition model, can perform object behavior recognition based on the deployed object behavior recognition model.

[0153] See Figure 5 , Figure 5 A flowchart of a model training method provided in an embodiment of the present invention, the method may include the following steps:

[0154] S501: Obtain the sample video and sample labels indicating the sample behavior of the sample objects in the sample video.

[0155] S502: Input the sample video into the video feature extraction network in the object behavior recognition model of the initial structure to obtain the global temporal and global spatial features of the sample video, which are used as the first sample features.

[0156] The object behavior recognition model also includes a first feature extraction network and a first large language model.

[0157] S503: Using the first feature extraction network, extract the image features of the image region occupied by the sample object in the sample video as the appearance features of the sample object.

[0158] S504: Perform feature fusion on the appearance features of the sample object and the features of the first sample to obtain the sample morphology adaptation features.

[0159] S505: Based on the first major language model, perform deep feature extraction on the features of the second sample to obtain the predicted behavior recognition result of the sample object.

[0160] The second sample feature is obtained based on the sample morphology adaptation feature.

[0161] S506: Based on the difference between the predicted behavior recognition result and the sample label, adjust the model parameters of the object behavior recognition model of the initial structure until the model converges, and obtain the trained object behavior recognition model.

[0162] Based on the above processing, the training device can obtain a more versatile object behavior recognition model; the trained object behavior recognition model is sent to the electronic device so that after the electronic device deploys the received object behavior recognition model, it can perform object behavior recognition based on the deployed object behavior recognition model, thereby improving the accuracy of the recognized object behavior.

[0163] Regarding step S501, in one approach, the video to be used acquired by the training device can be directly used as a sample video. For example, a user can edit a video already published online, and the user's editing result can be used as the video to be used; alternatively, the user can manually draw each video frame, and the video composed of the user-drawn video frames can be used as the video to be used. For example, a video to be released by the distributor of an entertainment work can be used as the video to be used. The video to be used can be in MP4 format, FLV format, etc., and this invention does not limit this. In this way, the probability of video content loss can be reduced, and the accuracy of the subsequently trained object behavior recognition model can be improved.

[0164] In another approach, the training device can preprocess the video to be used, and then use the preprocessed video as a sample video. Preprocessing includes at least one of the following: temporal sampling and spatial processing.

[0165] Specifically, the training device can first perform video decoding on the video to be used, converting it into a sequence of continuous video frames (which can be called original video frames). For example, the training device can perform video decoding based on FFmpeg technology. Temporal sampling involves sampling this sequence. For example, the training device can sample multiple video frames from the sequence at a preset sampling interval; or it can sample a specified number of video frames from the sequence. Then, the video composed of the sampled video frames is used as the sample video. The preset sampling interval and the specified number can be set based on the needs of the actual scenario. For example, the preset sampling interval can be 20 frames, and the specified number can be 32, 64, 128, etc., and this invention does not limit this. Spatial processing involves image compression of the video frames. For example, the specified size can be denoted as (H, W), where H represents the height of the compressed image and W represents the width of the compressed image. In actual scenarios, one of the specified sizes H and W is smaller than the original video frame, while the other is not larger than the original video frame. For example, the original video frame size can be (1280, 1280), the specified size can be (224, 224), or it can be written as 224×224.

[0166] In practical scenarios, the training device can first perform temporal sampling on the decoded sequence, and then perform spatial processing on the sampled video frames; the video composed of the spatially processed video frames is used as the sample video. The sample video can be represented in the form of a video frame tensor, which can be denoted as (N, C, H, W); N represents the aforementioned number, and C represents the number of color channels in each video frame. For example, when the video frame is in RGB format, C is 3.

[0167] Thus, by preprocessing the video to be used to obtain sample videos, the computational load of subsequent processing can be reduced, and the efficiency of object behavior recognition can be improved.

[0168] In another implementation, other devices can preprocess the video to be used in the manner described above to obtain sample videos, and then send the sample videos to the training device.

[0169] Regarding step S502, the object behavior recognition model of the initial structure includes a video feature extraction network for extracting video features from the sample video to obtain the global temporal and global spatial features of the sample video, which serve as the first sample features. The global temporal features represent the optical flow information between every two adjacent video frames in the sample video, and the global spatial features represent the image features of each video frame in the sample video. For example, the video feature extraction network can be C3D, TimeSFormer, or similar networks built based on Transformer. Alternatively, the video feature extraction network includes a temporal feature extraction network for extracting the global temporal features of the sample video, and a spatial feature extraction network for extracting the global spatial features of the sample video. The temporal feature extraction network can be RNN, LSTM, etc.; the spatial feature extraction network can be CNN, etc., and this invention does not limit this.

[0170] Taking TimeSFormer as an example, the training device can acquire the input data of the Transformer encoder in TimeSFormer in a similar manner to acquiring the input data of VIT. Specifically, for each video frame included in the sample video, the video frame is segmented into non-overlapping patches. The training device can generate a spatiotemporal location code indicating the temporal and spatial location of the image patch in the sample video. Then, the spatiotemporal location code corresponding to the image patch is added to the image patch. For example, for each image patch, a linear projection is used to map the pixel matrix of the image patch into a vector; the mapped vector and the spatiotemporal location code of the image patch are concatenated to obtain a feature concatenation result that retains the original position information of the image patch in the sample video. The feature concatenation result corresponding to each image patch is the input data of the Transformer encoder in TimeSFormer.

[0171] The Transformer encoder processes the feature stitching results corresponding to each image patch based on a multi-head self-attention mechanism. It can learn the global spatiotemporal dependencies between each image patch and then extract the features of each image patch, including the aforementioned global spatiotemporal dependencies, which are the first sample features of the sample video.

[0172] In one approach, to reduce the computational burden of subsequent processing, the training device can also perform pooling on the features output by the Transformer encoder, using the pooling result as the first sample feature of the sample video. Pooling methods could include max pooling, average pooling, etc.

[0173] Regarding step S503, the sample video includes at least one sample object. The sample object can be any object in the sample video, or it can be a specific object within the sample video. For example, it could be the object with the highest resolution in the sample video, or the object whose image area occupies the largest proportion in each video frame.

[0174] The object behavior recognition model also includes a first feature extraction network, used to extract image features of the image region (hereinafter referred to as the sample region) occupied by the sample object in the sample video. For example, for each video frame in the sample video, the sample region is the image region within the bounding rectangle of the sample object. Since the sample object is usually displayed in the video frame, it typically shows appearance information such as hair color, facial features, body proportions, and clothing. Therefore, extracting the image features of the sample region is equivalent to extracting the appearance features of the sample object in the sample region (hereinafter referred to as sample appearance features). The first feature extraction network may include an image region detection layer and a region feature extraction layer. For each video frame in the sample video, the image region detection layer is used to detect the image region occupied by the sample object in that video frame; then, the region feature extraction layer is used to extract the image features of the detected image region.

[0175] Regarding steps S504 and S505, after acquiring the sample appearance features and the first sample features, the object behavior recognition model can perform feature fusion on the sample appearance features and the first sample features. For example, the training device can perform feature concatenation on the sample appearance features and the first sample features according to feature channels to obtain sample morphology adaptation features.

[0176] The primary language model comprises multiple feature extraction layers for feature extraction. Using these multiple network layers for feature extraction is termed deep feature extraction. Since the feature dimensions of morphological adaptation features in real-world scenarios may not be supported by the primary language model, the training device can map these features to dimensions supported by the primary language model to obtain second sample features that can be processed by it. Then, deep feature extraction is performed on these second sample features based on the primary language model to obtain the predicted behavior recognition result, i.e., the behavior of the sample object in the sample video.

[0177] Regarding step S506, the training device can calculate a loss function value representing the difference between the predicted behavior recognition result and the sample label based on a preset loss function. The loss function value is then used to adjust the model parameters of the initial object behavior recognition model until the model convergence condition is met, resulting in a trained object behavior recognition model. For example, the preset loss function could be the MSE Loss (Mean Squared Error Loss) function, BCELoss (Binary Cross-Entropy Loss) function, etc.; the model convergence condition could be that the model parameters of the object recognition model have been adjusted a preset number of times, and the calculated loss value is less than a preset loss threshold, etc. This invention does not limit these limitations.

[0178] It is understood that the object behavior recognition model involved in this embodiment of the invention is an end-to-end model, that is, a sample video is input into the object recognition model, and the object recognition model outputs a predicted behavior recognition result. Accordingly, when adjusting the model parameters of the object behavior recognition model, the parameters of each network included in the object behavior recognition model will also be adjusted, specifically including the video feature extraction network, the first large language model, and the performance technique classification network and the second multilayer perceptron in subsequent embodiments. That is, the above-mentioned networks are jointly optimized to obtain a globally optimal trained object behavior recognition model.

[0179] In some embodiments, the first feature extraction network is an object detection network; the appearance features of the sample object are obtained by deep feature extraction of the sample video through the object detection network.

[0180] In this embodiment of the invention, the object detection network can be based on an object detection model, such as the feature extraction network in the YOLO model; or it can include feature extraction networks in other object detection models. The object detection model can include multiple feature extraction layers, and the object detection network can include some of these feature extraction layers. The object detection network processes the video to be identified, i.e., performs deep feature extraction on the video to be identified, to obtain the appearance features to be identified.

[0181] Based on the above processing, the training device can directly obtain the appearance features of the sample based on the object detection network, without having to first determine the image area occupied by the sample object in the sample video and then extract the image features of the determined image area. This simplifies the model structure, improves the efficiency of obtaining the appearance features of the sample, and thus improves the model training efficiency.

[0182] In some embodiments, during model training, the object behavior recognition model may further include a bounding box extraction network for extracting the image region (which can be called the predicted region) to which the sample object belongs from the sample video based on the appearance features of the sample object. The predicted region of each video frame in the sample video can be called the character bounding box sequence, and can be denoted as B. char ={b1, b2, ..., b T}. Among them, b x This represents the predicted region from the x-th video frame of the sample video. Correspondingly, the aforementioned sample labels can also indicate each sample region in the sample video. Experimental results show that by adjusting the model parameters of the object behavior recognition model based on the difference between the sample region and the predicted region, the accuracy of the target behavior recognition results obtained based on the trained object behavior recognition model will improve even after removing the detection box extraction network.

[0183] In some embodiments, the aforementioned step S504 may include the following steps:

[0184] Step E1: Divide the appearance features of the sample objects to obtain the sample scaling factor and sample bias factor.

[0185] Step E2: Scale the features of the first sample using the sample scaling factor to obtain the sample scaling result.

[0186] Step E3: Calculate the sum of the sample bias coefficient and the sample scaling result to obtain the sample morphological adaptation features.

[0187] In this embodiment of the invention, the sample appearance features are divided to obtain two coefficients. These two coefficients are used to adjust the first sample features, thereby achieving the fusion of the sample appearance features and the first sample features. Therefore, this embodiment of the invention does not limit the specific method of dividing the sample appearance features to obtain the two coefficients. For example, the sample appearance features can be divided into two matrices of the same size. For each matrix obtained, the matrix is ​​mapped to the target dimension to obtain a one-dimensional vector of the target dimension. Then, the two one-dimensional vectors obtained are used as the sample scaling coefficient and the sample bias coefficient, respectively. The matrices obtained from the division can be denoted as matrix A and matrix B; the one-dimensional vector obtained based on matrix A can be denoted as vector a, and the one-dimensional vector obtained based on matrix B can be denoted as vector b. Vector a can be used as the sample scaling coefficient, and vector b can be used as the sample bias coefficient; or, vector b can be used as the sample scaling coefficient, and vector a can be used as the sample bias coefficient, both of which are reasonable.

[0188] Furthermore, the features to be processed are dynamically adjusted using sample scaling and sample bias coefficients.

[0189] Based on the above processing, by dividing the sample appearance features into two coefficients and then using these two coefficients to adjust the first sample features, feature fusion of the first sample features and sample appearance features is achieved. That is, the adjustment method for the first sample features is guided by the sample appearance features. Subsequently, based on the obtained sample morphology adaptation features, the predicted behavior recognition results can be obtained, which can guide the object behavior recognition model to pay more attention to the information of the sample object itself, improving the accuracy of the subsequently trained object behavior recognition model.

[0190] In some embodiments, the object behavior recognition model further includes a first mapping network.

[0191] The aforementioned step E1 may include the following steps:

[0192] Step F1: Using the first mapping network, the appearance features of the sample object are mapped to the specified dimension to obtain the sample adaptation features.

[0193] The specified dimension is twice the feature dimension of the first sample feature.

[0194] Step F2: Divide the sample adaptation features into two vectors, which will be used as the sample scaling factor and the sample bias factor, respectively.

[0195] The feature dimension of each vector obtained by the partitioning is the same as the feature dimension of the first sample.

[0196] In this embodiment of the invention, the output dimension of the first mapping network is a specified dimension. The sample appearance features are mapped to the specified dimension through the first mapping network to obtain the sample adaptation features. The target dimension can be denoted as C, and correspondingly, the specified dimension can be denoted as 2×C. The sample adaptation features are one-dimensional vectors containing 2×C elements.

[0197] In some embodiments, the first mapping network includes a transformation network and a first multilayer perceptron.

[0198] The aforementioned step F1 may include the following steps:

[0199] Step G1: Using a transformation network, a self-attention mechanism is used to extract features from the appearance of the sample object to obtain the sample morphological features.

[0200] Step G2: Use the first multilayer perceptron to perform feature adaptation processing on the sample morphological features to obtain features of a specified dimension, which are used as sample adaptation features.

[0201] In this embodiment of the invention, the transformation network can be a network built based on a self-attention mechanism, such as a Transformer encoder. The sample appearance features are input into the transformation network, which uses a self-attention mechanism to extract features from the sample appearance features. This can further remove redundant features (such as features of the background region in the sample area) from the sample appearance features, resulting in more accurate sample appearance features, which are then used as sample morphology features.

[0202] Then, a first multilayer perceptron is used to perform feature adaptation processing on the sample morphological features. Specifically, the first multilayer perceptron may include an input layer, a hidden layer, and an output layer. The input layer receives the sample morphological features as input data to the first multilayer perceptron. The hidden layer may include at least one fully connected layer, and each fully connected layer may be followed by a non-linear activation function, such as ReLU. The fully connected layer in the hidden layer is used to extract the non-linear features in the sample morphological features. The output layer includes a fully connected layer, and this fully connected layer does not include an activation function; the number of output channels of this fully connected layer is 2×C. The output layer is used to process the non-linear features extracted by the hidden layer. After the first multilayer perceptron performs a complete forward operation, the output layer can output a one-dimensional vector containing 2×C elements, thus obtaining the sample adaptation features of the specified dimension.

[0203] Then, the sample adaptation feature is divided into two equal-length parts, resulting in two one-dimensional vectors. These two one-dimensional vectors can be used as the sample scaling factor and the sample bias factor, respectively. For example, the vector consisting of the first C elements of the sample adaptation feature can be used as the sample scaling factor, and the vector consisting of the remaining C elements can be used as the sample bias factor; alternatively, the vector consisting of the first C elements of the sample adaptation feature can be used as the sample bias factor, and the vector consisting of the remaining C elements can be used as the sample scaling factor; alternatively, the vector consisting of the odd-numbered elements of the sample adaptation feature can be used as the sample scaling factor, and the vector consisting of the even-numbered elements can be used as the sample bias factor; alternatively, the vector consisting of the odd-numbered elements of the sample adaptation feature can be used as the sample bias factor, and the vector consisting of the even-numbered elements can be used as the sample scaling factor. All of these are reasonable.

[0204] Based on the above processing, while making full use of the appearance features of the samples, two coefficients can be obtained through a single mapping, which can simplify the model structure of the object behavior recognition model, reduce the amount of computation to obtain the two coefficients, improve the computational efficiency of calculating the two coefficients, and improve the model training efficiency.

[0205] In some embodiments, the object behavior recognition model further includes a performance technique classification network and a second multilayer perceptron.

[0206] Prior to step S505, the method may further include the following steps:

[0207] Step H1: Input the features of the first sample into the performance technique classification network to obtain the technique category representing the performance technique of the video content of the sample video.

[0208] Step H2: Concatenate the first sample feature and the label of the method category corresponding to the first sample feature to obtain the sample concatenation result.

[0209] Step H3: The sample splicing results are processed by a second multilayer perceptron to obtain sample technique adaptation features.

[0210] Step H4: Perform feature fusion on the sample morphology adaptation features and sample technique adaptation features to obtain the second sample features.

[0211] In this embodiment of the invention, a representation method classification network can also be used to obtain the method category representing the video content representation method of the sample video. The video content representation method of the sample video can indicate the way the sample object is displayed in the sample video.

[0212] The performance technique classification network is used to determine the performance technique category (hereinafter referred to as the sample technique category) of the video content that matches the features of the first sample from various preset technique categories. The performance technique classification network can be a VIT model, or it can be other classification models. This invention does not limit this.

[0213] The preset technique categories can be set by technicians based on their work experience. For example, they can include the aforementioned "speed line appears", "extreme facial distortion", "shock wave", "magic release", "character's whole body is on fire", "transformation", or other technique categories that may appear in actual scenarios. This invention does not limit these categories.

[0214] The first sample feature and its corresponding technique category identifier are concatenated, for example, by feature channel, to obtain the sample concatenation result. Then, a second multilayer perceptron performs feature adaptation processing on the sample concatenation result to obtain sample technique adaptation features that indicate the technique category corresponding to the first sample feature and the sample video. In this way, the performance technique is converted into semantic information that is beneficial to the understanding of the object behavior recognition model.

[0215] Furthermore, the training device can perform feature fusion on sample morphological adaptation features and sample technique adaptation features. For example, it can perform feature splicing on sample morphological adaptation features and sample technique adaptation features, map the feature splicing results according to the dimensions supported by the first major language model, and use the mapping results as the second sample features.

[0216] In some embodiments, the object behavior recognition model further includes a second mapping network and a second major language model.

[0217] The aforementioned step H4 may include the following steps:

[0218] Step I1: Perform feature fusion on the sample morphology adaptation features and sample technique adaptation features to obtain the first sample fusion feature.

[0219] Step I2: Using the second mapping network, the first sample fusion features are mapped according to the dimensions supported by the second largest language model to obtain the second sample fusion features.

[0220] Step I3: Perform cross-modal alignment on the fusion features of the second sample to obtain sample prompt information supported by the second largest language model.

[0221] Step I4: Use the second language model to perform deep feature extraction on the sample prompt information to obtain the second sample features.

[0222] In this embodiment of the invention, the training device performs feature fusion on sample morphology adaptation features and sample technique adaptation features (hereinafter collectively referred to as adaptation features). For example, after concatenating the two adaptation features, a Transformer is used to extract features from the concatenated result, and the extracted feature result is used as the first sample fusion feature; or, the training device can map any adaptation feature to a specified dimension, then divide the mapping result into two coefficients, where the feature dimensions of both coefficients are the target dimension, and use the divided coefficients to adjust another adaptation feature, using the adjusted feature extraction result as the first sample fusion feature. Both of these are reasonable. The first sample fusion feature can be denoted as F. input_visual Then, a second mapping network is used for feature alignment to adjust F... input_visual Mapping to the dimension supported by the second largest language model, the mapping result can be denoted as F'. input_visual Only then can the second largest language model be used to process the results based on the mapping.

[0223] Furthermore, based on Q-former, vision-language fusion can be performed. Specifically, the training device can be configured with a learnable query vector Q. learnable Before performing cross-modal alignment, Q learnable The parameters in Q can be preset, such as all being 0. A cross-attention mechanism is used to apply attention to Q. learnable and F' input_visual Perform feature extraction and adjust Q using the extracted features. learnable The parameters in the adjusted Q learnable Q, which has already undergone one learning session learnable Then, continuing based on the Q that has already been learned once... learnable and F' input_visual Perform feature extraction and indicate when the stopping condition is met, such as Q. learnable After a predetermined number of learning iterations, the latest obtained Q... learnable As a result of cross-modal alignment, it can be denoted as F qureied_visual F qureied_visual The second largest visual feature that a language model can process (i.e., target cue information) can also be called soft cue.

[0224] Furthermore, a second language model is used to extract deep features from the sample prompts. This second language model can be the LLM part of a pre-trained MLLM that has learned pre-trained knowledge, such as MiniCPMV-2.6. During pre-training, the MLLM can learn scene information (such as battlefield, school, park, etc.) and character dialogue. Using the pre-trained knowledge and reasoning ability of the second language model, semantic features containing semantic information are extracted from the sample prompts that integrate visual and semantic information.

[0225] Based on the above processing, the object behavior recognition model fuses the adapted visual features and adapted semantic features to obtain second sample features. Subsequently, the predicted behavior recognition result is obtained based on these second sample features; that is, semantic features are used to assist in the behavior recognition process. Compared to using only visual features to recognize the behavior of sample objects, using the behavior indicated by semantic features allows the object behavior recognition model to interpret visual features more deeply, resulting in more accurate sample behavior recognition results. This can subsequently improve the accuracy of the trained object recognition model.

[0226] In some embodiments, the correspondence between video content and technique categories can be manually set by technicians. The training device can then find the technique category corresponding to the video content of a sample video from this correspondence, and use it as the sample technique category.

[0227] In some embodiments, the aforementioned step S505 may include the following steps:

[0228] Step J1: Using the deep feature extraction layer in the first language model, perform deep feature extraction on the features of the second sample to obtain the features of the third sample.

[0229] Step J2: Calculate the matching probability between the features of the third sample and each behavioral label using the label classification layer in the first language model.

[0230] Step J3: Take the behavior indicated by the behavior label that has a matching probability greater than the third sample feature among all behavior labels as the first predicted behavior recognition result of the sample object.

[0231] In this embodiment of the invention, the deep feature extraction layer, i.e., the feature extraction layer of the first large language model in the aforementioned embodiment, is used to perform deep feature extraction on the second sample features. The training device can directly use the deep feature extraction layer in the first large language model to perform deep feature extraction on the second sample features to obtain the third sample features; or, to reduce the amount of computation and improve the efficiency of object behavior recognition, the second sample features can be pooled first to obtain pooled features, and then deep feature extraction can be performed on the pooled features to obtain the third sample features.

[0232] The label classification layer in the first language model may include at least one fully connected layer, and each fully connected layer may be followed by an activation function, such as a sigmoid function, to calculate the matching probability between the third sample feature and a preset M behavioral labels. A behavioral label indicates a type of behavior, such as the aforementioned "fighting," "transformation," and "contemplation." Behavioral labels can be set by technicians based on their work experience. This invention does not limit the number of behavioral labels, nor does it limit the specific behavior indicated by the behavioral labels.

[0233] The label classification layer can output a probability vector, denoted as P = {p1, p2, p3, ..., pm}; px represents the matching probability between the third sample feature and the preset x-th behavior label.

[0234] Furthermore, the training device can identify behavior labels from among the behavior labels whose matching probability with the third sample feature is greater than a preset probability threshold, and use these as predicted behavior labels. The behavior indicated by the predicted behavior label is used as the first predicted behavior recognition result of the sample object. The preset probability threshold can be set by technicians based on their work experience, such as 0.8. In actual scenarios, the matching probability of each behavior label with the third sample feature may not be greater than the preset probability threshold, or there may be multiple behavior labels whose matching probability with the third sample feature may be greater than the preset probability threshold.

[0235] The preset behavioral labels can be called a closed set. Based on the embodiments of the present invention, the object behavior recognition model can determine the behavior of the sample object from the behaviors already included in the closed set.

[0236] In some embodiments, after step J2, the method may further include the following steps: using the text output layer in the first large language model to decode the features of the third sample to obtain a second predicted behavior recognition result indicating the behavior of the sample object.

[0237] In this embodiment of the invention, since the behavioral labels included in the aforementioned closed set are limited and difficult to cover all the behaviors of objects that may appear in the actual scene, in order to reduce the probability that the behavior of the sample object is not detected, the training device can also input descriptive text into the object behavior recognition model to instruct the object behavior recognition model to use natural language to describe the behaviors not included in the closed set. For example, the descriptive text can be "Please describe the unrecognized object behavior in the video".

[0238] The text output layer in the first language model acts as a decoder for decoding the features extracted by the first language model. The training device converts the aforementioned descriptive text into a token, and inputs the token and the second sample features into the first language model. Correspondingly, the deep feature extraction layer in the first language model can perform deep feature extraction on the token to obtain token features; the text output layer in the first language model decodes the token features and the third sample features to obtain the output text of the first language model, which serves as the result of the second predicted behavior recognition. The output text can be called the open set.

[0239] In real-world scenarios, the first language model can also output indications that the sample video contains behaviors not included in a closed set. For example, the first language model can output "Unrecognized object behaviors in the video include: arguing." The text before "arguing" is the aforementioned indication information.

[0240] In some embodiments, the electronic device can input the aforementioned token features and the aforementioned sample prompt information into the second language model. By using the aforementioned descriptive text to assist the second language model in deep feature extraction, the second language model can be guided to extract more accurate second sample features, thereby improving the accuracy of the subsequent predicted behavior recognition results and thus improving the accuracy of the trained object behavior recognition model.

[0241] In some embodiments, to further improve the accuracy of target behavior recognition results obtained by subsequently using the trained object behavior recognition model, the training device can use a finer-grained training set (including sample videos and corresponding sample labels) to train the object behavior recognition model. For example, based on the actual business scenario requirements for which the object behavior recognition model will be used, a training set for that business scenario (which can be called a specific business training set) can be determined, and then the object behavior recognition model can be trained using the specific business training set. For example, in an actual business scenario, if the object behavior recognition model needs to be used to recognize the behavior of anime characters, the specific business training set may include: sample videos of anime characters "transforming," and sample labels indicating that the behavior of anime characters is "transformation"; sample videos of anime characters "mecha-ifying," and sample labels indicating that the behavior of anime characters is "mecha-ifying"; sample videos of anime characters "releasing magic," and sample labels indicating that the behavior of anime characters is "releasing magic," etc.

[0242] Based on the same inventive concept as the object behavior recognition method described above, embodiments of the present invention also provide an object behavior recognition device. See [link to related document]. Figure 6 , Figure 6This is a structural diagram of an object behavior recognition device provided in an embodiment of the present invention. The device includes: a first acquisition module 601, used to acquire a video to be recognized; a first feature extraction module 602, used to input the video to be recognized into a video feature extraction network in a pre-trained object behavior recognition model to obtain global temporal features and global spatial features of the video to be recognized as features to be processed; wherein, the object behavior recognition model further includes a first feature extraction network and a first large language model; the pre-trained object behavior recognition model is trained based on sample videos and sample labels indicating sample behaviors of sample objects in the sample videos; a second feature extraction module 603, used to extract image features of the image region occupied by the object to be recognized in the video to be recognized using the first feature extraction network as appearance features of the object to be recognized; a first fusion module 604, used to perform feature fusion on the appearance features of the object to be recognized and the features to be processed to obtain object morphology adaptation features; and a recognition result acquisition module 605, used to perform deep feature extraction on the features to be recognized based on the first large language model to obtain the target behavior recognition result of the object to be recognized; wherein, the features to be recognized are obtained based on the object morphology adaptation features.

[0243] Optionally, the first fusion module 604 is specifically used to: divide the appearance features of the object to be identified to obtain a target scaling factor and a target bias factor; use the target scaling factor to scale the features to be processed to obtain a target scaling result; calculate the sum of the target bias factor and the target scaling result to obtain the object shape adaptation feature.

[0244] Optionally, the object behavior recognition model further includes a first mapping network; the first fusion module 604 is specifically used to: use the first mapping network to map the appearance features of the object to be recognized to a specified dimension to obtain target adaptation features; wherein, the specified dimension is twice the feature dimension of the feature to be processed; divide the target adaptation features into two vectors, which are respectively used as target scaling coefficient and target bias coefficient; wherein, the feature dimension of each vector obtained by division is the same as the feature dimension of the feature to be processed.

[0245] Optionally, the first mapping network includes a transformation network and a first multilayer perceptron; the first fusion module 604 is specifically used to: use the transformation network to extract features from the appearance features of the object to be identified using a self-attention mechanism to obtain object morphological features; use the first multilayer perceptron to perform feature adaptation processing on the object morphological features to obtain features of a specified dimension, which are used as target adaptation features.

[0246] Optionally, the object behavior recognition model further includes a performance technique classification network and a second multilayer perceptron; the device further includes: a cross-modal fusion module, used to, before the recognition result acquisition module 605 performs deep feature extraction based on the first large language model to obtain the target behavior recognition result of the object to be recognized, input the features to be processed into the performance technique classification network to obtain the technique category representing the video content performance technique of the video to be recognized; concatenate the features to be processed and the identifiers of the technique categories corresponding to the features to be processed to obtain a target concatenation result; perform feature adaptation processing on the target concatenation result through the second multilayer perceptron to obtain performance technique adaptation features; and perform feature fusion on the object morphology adaptation features and the performance technique adaptation features to obtain the features to be recognized.

[0247] Optionally, the object behavior recognition model further includes a second mapping network and a second major language model; the cross-modal fusion module is specifically used for: fusing the object morphology adaptation features and the performance technique adaptation features to obtain a first visual fusion feature; using the second mapping network, mapping the first visual fusion feature according to the dimensions supported by the second major language model to obtain a second visual fusion feature; performing cross-modal alignment on the second visual fusion feature to obtain target prompt information supported by the second major language model; and using the second major language model to perform deep feature extraction on the target prompt information to obtain the features to be recognized.

[0248] Optionally, the recognition result acquisition module 605 is specifically used to: perform deep feature extraction on the feature to be identified using the deep feature extraction layer in the first large language model to obtain the feature to be used; calculate the matching probability between the feature to be used and each behavior label using the label classification layer in the first large language model; and take the behavior indicated by the behavior label with a matching probability greater than a preset probability threshold as the first target behavior recognition result of the object to be identified.

[0249] Optionally, the device further includes a target decoding module, configured to, after the recognition result acquisition module 605 performs deep feature extraction on the feature to be recognized using the deep feature extraction layer in the first large language model to obtain the feature to be used, perform decoding on the feature to be used using the text output layer in the first large language model to obtain a second target behavior recognition result indicating the behavior of the object to be recognized.

[0250] Optionally, the first feature extraction network is an object detection network; the appearance features of the object to be identified are obtained by deep feature extraction of the video to be identified through the object detection network.

[0251] Based on the same inventive concept as the model training method described above, this embodiment of the invention also provides a model training apparatus. See also... Figure 7 , Figure 7 This is a structural diagram of a model training apparatus provided in an embodiment of the present invention. The apparatus includes: a second acquisition module 701, used to acquire sample videos and sample labels indicating sample behaviors of sample objects in the sample videos; a third feature extraction module 702, used to input the sample videos into a video feature extraction network in an initial structure object behavior recognition model to obtain global temporal features and global spatial features of the sample videos as first sample features; wherein, the object behavior recognition model further includes a first feature extraction network and a first large language model; and a fourth feature extraction module 703, used to extract the image region occupied by the sample objects in the sample videos using the first feature extraction network. The first sample feature is used as the appearance feature of the sample object; the second fusion module 704 is used to fuse the appearance feature of the sample object and the first sample feature to obtain the sample morphology adaptation feature; the prediction result acquisition module 705 is used to perform deep feature extraction on the second sample feature based on the first large language model to obtain the predicted behavior recognition result of the sample object; wherein, the second sample feature is obtained based on the sample morphology adaptation feature; the parameter adjustment module 706 is used to adjust the model parameters of the initial structure object behavior recognition model based on the difference between the predicted behavior recognition result and the sample label until the model converges to obtain the trained object behavior recognition model.

[0252] Optionally, the second fusion module 704 is specifically used for: dividing the appearance features of the sample object to obtain a sample scaling coefficient and a sample bias coefficient; scaling the first sample feature using the sample scaling coefficient to obtain a sample scaling result; and calculating the sum of the sample bias coefficient and the sample scaling result to obtain the sample morphology adaptation feature.

[0253] Optionally, the object behavior recognition model further includes a first mapping network; the second fusion module 704 is specifically used to: use the first mapping network to map the appearance features of the sample object to a specified dimension to obtain sample adaptation features; wherein, the specified dimension is twice the feature dimension of the first sample feature; divide the sample adaptation features into two vectors, which are respectively used as sample scaling coefficient and sample bias coefficient; wherein, the feature dimension of each vector obtained by division is the same as the feature dimension of the first sample feature.

[0254] Optionally, the first mapping network includes a transformation network and a first multilayer perceptron; the second fusion module 704 is specifically used to: use the transformation network to extract features from the appearance features of the sample object using a self-attention mechanism to obtain sample morphological features; use the first multilayer perceptron to perform feature adaptation processing on the sample morphological features to obtain features of a specified dimension, which are used as sample adaptation features.

[0255] Optionally, the object behavior recognition model further includes a performance technique classification network and a second multilayer perceptron; the device further includes: a sample fusion module, used to, before the prediction result acquisition module 705 performs deep feature extraction on the second sample features based on the first large language model to obtain the prediction recognition result of the sample object, input the first sample features into the performance technique classification network to obtain the technique category representing the video content performance technique of the sample video; concatenate the first sample features and the identifier of the technique category corresponding to the first sample features to obtain a sample concatenation result; perform feature adaptation processing on the sample concatenation result through the second multilayer perceptron to obtain sample technique adaptation features; and perform feature fusion on the sample morphology adaptation features and the sample technique adaptation features to obtain the second sample features.

[0256] Optionally, the object behavior recognition model further includes a second mapping network and a second major language model; the sample fusion module is specifically used for: fusing the sample morphology adaptation features and the sample technique adaptation features to obtain a first sample fusion feature; using the second mapping network, mapping the first sample fusion feature according to the dimensions supported by the second major language model to obtain a second sample fusion feature; performing cross-modal alignment on the second sample fusion feature to obtain sample prompt information that the second major language model supports processing; and using the second major language model to perform deep feature extraction on the sample prompt information to obtain a second sample feature.

[0257] Optionally, the prediction result acquisition module 705 is specifically used for: using the deep feature extraction layer in the first large language model to perform deep feature extraction on the second sample features to obtain the third sample features; using the label classification layer in the first large language model to calculate the matching probability between the third sample features and each behavior label; and taking the behavior indicated by the behavior label whose matching probability with the third sample features is greater than a preset probability threshold as the first predicted behavior recognition result of the sample object.

[0258] Optionally, the apparatus further includes a sample decoding module, configured to, after the prediction result acquisition module 705 performs deep feature extraction on the second sample features using the deep feature extraction layer in the first large language model to obtain the third sample features, perform decoding on the third sample features using the text output layer in the first large language model to obtain a second predicted behavior recognition result indicating the behavior of the sample object.

[0259] Optionally, the first feature extraction network is an object detection network; the appearance features of the sample object are obtained by deep feature extraction of the sample video through the object detection network.

[0260] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804. The memory 803 is used to store computer programs. When the processor 801 executes the program stored in the memory 803, it implements any of the object behavior recognition method steps described in the foregoing embodiments, or implements any of the model training method steps described in the foregoing embodiments.

[0261] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0262] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0263] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0264] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0265] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the object behavior recognition method described in any of the above embodiments, or implements the model training method described in any of the above embodiments.

[0266] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the object behavior recognition methods described in the above embodiments, or to execute any of the model training methods described in the above embodiments.

[0267] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0268] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0269] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0270] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method of recognizing behavior of an object, characterized by, The method comprises: acquiring a video to be identified; inputting the video to be identified into a video feature extraction network in a pre-trained object behavior identification model to obtain global time domain features and global space features of the video to be identified as to-be-processed features; wherein the object behavior identification model further comprises a first feature extraction network and a first large language model; extracting image features of an image region occupied by an object to be identified in the video to be identified as appearance features of the object to be identified by using the first feature extraction network; performing feature fusion on the appearance features of the object to be identified and the to-be-processed features to obtain object form adaptation features; performing deep feature extraction on to-be-identified features based on the first large language model to obtain a target behavior identification result of the object to be identified; wherein the to-be-identified features are obtained based on the object form adaptation features.

2. The method of claim 1, wherein, The method further comprises: dividing the appearance features of the object to be identified to obtain a target scaling coefficient and a target bias coefficient; scaling the to-be-processed features using the target scaling coefficient to obtain a target scaling result; calculating a sum value of the target bias coefficient and the target scaling result to obtain object form adaptation features.

3. The method of claim 2, wherein, The object behavior identification model further comprises a first mapping network; The method further comprises: mapping the appearance features of the object to be identified to a specified dimension using the first mapping network to obtain target adaptation features; wherein the specified dimension is twice the feature dimension of the to-be-processed features; dividing the target adaptation features into two vectors as a target scaling coefficient and a target bias coefficient; wherein the feature dimension of each vector obtained by division is the same as the feature dimension of the to-be-processed features.

4. The method of claim 3, wherein, The first mapping network comprises a conversion network and a first multi-layer perception; The method further comprises: using a self-attention mechanism to perform feature extraction on the appearance features of the object to be identified using the conversion network to obtain object form features; using the first multi-layer perception to perform feature adaptation processing on the object form features to obtain features of the specified dimension as target adaptation features.

5. The method of claim 2, wherein, The object behavior identification model further comprises a performance method classification network and a second multi-layer perception; Before performing deep feature extraction on to-be-identified features based on the first large language model to obtain a target behavior identification result of the object to be identified, the method further comprises: inputting the to-be-processed features into the performance method classification network to obtain a method category representing a video content performance method of the video to be identified; splicing the to-be-processed features and an identifier of the method category corresponding to the to-be-processed features to obtain a target splicing result; performing feature adaptation processing on the target splicing result by the second multi-layer perception to obtain performance method adaptation features; The object form adaptation feature and the performance method adaptation feature are fused to obtain a to-be-recognized feature.

6. The method of claim 5, wherein, The object behavior recognition model further comprises a second mapping network and a second large language model; The object form adaptation feature and the performance method adaptation feature are fused to obtain a to-be-recognized feature, comprising: The object form adaptation feature and the performance method adaptation feature are fused to obtain a first visual fusion feature; The first visual fusion feature is mapped according to dimensions supported by the second large language model by using the second mapping network to obtain a second visual fusion feature; The second visual fusion feature is cross-modality aligned to obtain target prompt information supported by the second large language model for processing; The target prompt information is subjected to deep feature extraction by using the second large language model to obtain a to-be-recognized feature.

7. The method of claim 5, wherein, The first large language model is used to perform deep feature extraction on the to-be-recognized feature to obtain a target behavior recognition result of the to-be-recognized object, comprising: The first large language model is used to perform deep feature extraction on the to-be-recognized feature to obtain a to-be-utilized feature; The first large language model is used to calculate a matching probability of the to-be-utilized feature with each behavior label; The behavior indicated by the behavior label whose matching probability with the to-be-utilized feature is greater than a preset probability threshold is taken as a first target behavior recognition result of the to-be-recognized object.

8. The method of claim 7, wherein, After the first large language model is used to perform deep feature extraction on the to-be-recognized feature to obtain a to-be-utilized feature, the method further comprises: The first large language model is used to decode the to-be-utilized feature to obtain a second target behavior recognition result indicating a behavior of the to-be-recognized object.

9. The method of claim 1, wherein, The first feature extraction network is a target detection network; The appearance feature of the to-be-recognized object is obtained by deep feature extraction of the to-be-recognized video by the target detection network.

10. A model training method, comprising: The method comprises: A sample video and a sample label indicating a sample behavior of a sample object in the sample video are obtained; The sample video is input into a video feature extraction network in an object behavior recognition model of an initial structure to obtain global time domain features and global spatial features of the sample video as first sample features; wherein the object behavior recognition model further comprises a first feature extraction network and a first large language model; The first feature extraction network is used to extract image features of an image region occupied by a sample object in the sample video as an appearance feature of the sample object; The appearance feature of the sample object and the first sample feature are fused to obtain a sample form adaptation feature; A second sample feature is subjected to deep feature extraction by the first large language model to obtain a predicted behavior recognition result of the sample object; wherein the second sample feature is obtained based on the sample form adaptation feature. Based on the difference between the predicted behavior recognition result and the sample label, the model parameters of the object behavior recognition model of the initial structure are adjusted until the model converges, and a trained object behavior recognition model is obtained.

11. The method of claim 10, wherein, The appearance feature of the sample object and the first sample feature are fused to obtain a sample shape adaptive feature, including: The appearance feature of the sample object is divided to obtain a sample scaling coefficient and a sample bias coefficient; The first sample feature is scaled using the sample scaling coefficient to obtain a sample scaling result; The sum of the sample bias coefficient and the sample scaling result is calculated to obtain a sample shape adaptive feature.

12. The method of claim 11, wherein, The object behavior recognition model further includes a first mapping network; The appearance feature of the sample object is divided to obtain a sample scaling coefficient and a sample bias coefficient, including: The appearance feature of the sample object is mapped to a specified dimension using the first mapping network to obtain a sample adaptive feature; wherein the specified dimension is twice the feature dimension of the first sample feature; The sample adaptive feature is divided into two vectors, which are used as the sample scaling coefficient and the sample bias coefficient, respectively; wherein the feature dimension of each vector obtained by division is the same as the feature dimension of the first sample feature.

13. The method of claim 12, wherein, The first mapping network includes a conversion network and a first multilayer perception; The appearance feature of the sample object is mapped to a specified dimension using the first mapping network to obtain a sample adaptive feature, including: The appearance feature of the sample object is extracted using a self-attention mechanism using the conversion network to obtain a sample shape feature; The sample shape feature is processed using the first multilayer perception to obtain a feature of a specified dimension as a sample adaptive feature.

14. The method of claim 11, wherein, The object behavior recognition model further includes a performance method classification network and a second multilayer perception; Before the first large language model is used to extract deep features from the second sample feature to obtain the predicted recognition result of the sample object, the method further includes: The first sample feature is input into the performance method classification network to obtain a method category representing the video content performance method of the sample video; The first sample feature and the identifier of the method category corresponding to the first sample feature are spliced to obtain a sample splicing result; The sample splicing result is processed using the second multilayer perception to obtain a sample method adaptive feature; The sample shape adaptive feature and the sample method adaptive feature are fused to obtain a second sample feature.

15. The method of claim 14, wherein, The object behavior recognition model further includes a second mapping network and a second large language model; The sample shape adaptive feature and the sample method adaptive feature are fused to obtain a second sample feature, including: The sample shape adaptive feature and the sample method adaptive feature are fused to obtain a first sample fusion feature; The first sample fusion feature is mapped according to the dimension supported by the second large language model using the second mapping network to obtain a second sample fusion feature; The second sample fusion feature is cross-modal aligned to obtain sample prompt information supported by the second large language model for processing; Deep feature extraction is performed on the sample prompt information by using the second large language model to obtain a second sample feature.

16. The method of claim 14, wherein, Deep feature extraction is performed on the second sample feature based on the first large language model to obtain a prediction behavior recognition result of the sample object, including: Deep feature extraction is performed on the second sample feature by using a deep feature extraction layer in the first large language model to obtain a third sample feature; The matching probability of the third sample feature with each behavior label is calculated by using a label classification layer in the first large language model; The behavior indicated by the behavior label with a matching probability greater than a preset probability threshold in each behavior label is taken as a first prediction behavior recognition result of the sample object.

17. The method of claim 16, wherein, After deep feature extraction is performed on the second sample feature by using the deep feature extraction layer in the first large language model to obtain a third sample feature, the method further includes: The third sample feature is decoded by using a text output layer in the first large language model to obtain a second prediction behavior recognition result indicating the behavior of the sample object.

18. The method of claim 10, wherein, The first feature extraction network is a target detection network. The appearance feature of the sample object is obtained by deep feature extraction on the sample video by using the target detection network.

19. An object behavior recognition apparatus, characterized by comprising: The device includes: A first acquisition module is configured to acquire a video to be recognized. A first feature extraction module is configured to input the video to be recognized into a video feature extraction network in a pre-trained object behavior recognition model to obtain global time domain features and global spatial features of the video to be recognized as to-be-processed features. A second feature extraction module is configured to extract image features of an image region occupied by an object to be recognized in the video to be recognized as an appearance feature of the object to be recognized by using the first feature extraction network. A first fusion module is configured to perform feature fusion on the appearance feature of the object to be recognized and the to-be-processed features to obtain object morphology adaptive features. An identification result acquisition module is configured to perform deep feature extraction on to-be-recognized features based on the first large language model to obtain a target behavior recognition result of the object to be recognized, wherein the to-be-recognized features are obtained based on the object morphology adaptive features.

20. A model training apparatus, comprising: The device includes: A second acquisition module is configured to acquire a sample video and a sample label indicating a sample behavior of a sample object in the sample video. A third feature extraction module is configured to input the sample video into a video feature extraction network in an object behavior recognition model with an initial structure to obtain global time domain features and global spatial features of the sample video as first sample features. The object behavior recognition model further includes a first feature extraction network and a first large language model. The fourth feature extraction module is configured to extract image features of an image region occupied by the sample object in the sample video by using the first feature extraction network, as appearance features of the sample object; The second fusion module is configured to perform feature fusion on the appearance features of the sample object and the first sample features, to obtain sample morphological adaptation features; The prediction result acquisition module is configured to perform deep feature extraction on the second sample features based on the first large language model, to obtain a prediction behavior recognition result of the sample object, wherein the second sample features are obtained based on the sample morphological adaptation features; The parameter adjustment module is configured to adjust model parameters of the object behavior recognition model with the initial structure based on a difference between the prediction behavior recognition result and the sample label, until the model converges, to obtain a trained object behavior recognition model.

21. An electronic device, comprising: The apparatus includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory, to implement the method of any one of claims 1-9 or claims 10-18.

22. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and when the computer program is executed by the processor, the method steps of any one of claims 1-9 or claims 10-18 are implemented.

Images