Video behavior recognition method, device and equipment based on multimodal large model fine-tuning

Abstract

Embodiments of the present disclosure relate to the field of video detection technology, and specifically to a video behavior recognition method, device, and equipment based on fine-tuning of a multimodal large model. A specific implementation of the method disclosed herein includes: obtaining a video behavior recognition data set and an initial video behavior recognition model; for each video behavior recognition data, performing the following steps: determining the key frame image corresponding to the video frame sequence included in the video behavior recognition data; storing the key frame image to a preset image storage location; generating training samples based on a label text template, key frame images, image storage path, and behavior labels included in the video behavior recognition data, wherein the training samples include sample images and sample label data; training the initial video behavior recognition model, and determining it as a video behavior recognition model; inputting the target video data into the video behavior recognition model to obtain video behavior text information. This implementation can increase the training time of the video behavior recognition model.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of video detection technology, and in particular to a video behavior recognition method, device, and apparatus based on fine-tuning of a multimodal large model. Background Art

[0002] Video behavior recognition is a technology used to identify actions in video clips. At present, the common method for video behavior recognition is: first, the original video data set is directly used as a training sample set to train the multimodal video behavior recognition model with all parameters. Then, the trained video behavior recognition model is used to perform behavior recognition on the video data to be detected, and the recognized behavior category information is output.

[0003] However, it is found in practice that when the above method is adopted, the following technical problems often occur:

[0004] First, video data is usually large in size and takes a long time to process. If the video behavior recognition model is trained directly based on the original video data set, the training time of the video behavior recognition model will be too long.

[0005] Second, if all parameters of the multimodal video behavior recognition model are trained, it will consume a lot of computing resources and time, resulting in excessive use of computing resources and a long model training time, which makes it difficult to use it for behavior recognition on the video data to be detected in a timely manner.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose video behavior recognition methods, devices and equipment based on fine-tuning of a multimodal large model to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a video behavior recognition method based on multimodal large model fine-tuning, the method comprising: obtaining a video behavior recognition data set and an initial video behavior recognition model, wherein each video behavior recognition data in the above video behavior recognition data set includes a video frame sequence and a behavior label, and the above initial video behavior recognition model includes a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module and a text vector embedding module; for each video behavior recognition data in the above video behavior recognition data set, performing the following steps: determining a key frame image corresponding to the video frame sequence included in the above video behavior recognition data; storing the key frame image in a preset image storage location to generate an image storage path; generate training samples based on a preset label text template, the key frame image, the image storage path and the behavior label included in the video behavior recognition data, wherein the training samples include sample images and sample label data; train an initial video behavior recognition model based on the generated training samples, and determine the trained initial video behavior recognition model as a video behavior recognition model; in response to receiving behavior recognition request information for target video data, input the target video data into the video behavior recognition model to obtain video behavior text information, and send the video behavior text information to a user terminal corresponding to the behavior recognition request information.

[0010] In a second aspect, some embodiments of the present disclosure provide a video behavior recognition device based on multimodal large model fine-tuning, the device comprising: an acquisition unit, configured to acquire a video behavior recognition data set and an initial video behavior recognition model, wherein each video behavior recognition data in the above video behavior recognition data set includes a video frame sequence and a behavior label, and the above initial video behavior recognition model includes a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module and a text vector embedding module; an execution unit, configured to perform the following steps for each video behavior recognition data in the above video behavior recognition data set: determining a key frame image corresponding to the video frame sequence included in the above video behavior recognition data; storing the key frame image to a preset image storage location; storing the key frame image in ... Generate an image storage path; generate training samples based on a preset label text template, the key frame image, the image storage path and the behavior label included in the video behavior recognition data, wherein the training samples include sample images and sample label data; a training and determination unit is configured to train an initial video behavior recognition model based on each generated training sample, and determine the trained initial video behavior recognition model as a video behavior recognition model; an input and sending unit is configured to input the target video data into the video behavior recognition model in response to receiving behavior recognition request information for target video data, obtain video behavior text information, and send the video behavior text information to a user terminal corresponding to the behavior recognition request information.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0013] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the video behavior recognition method based on multimodal large model fine-tuning of some embodiments of the present disclosure, the training time of the video behavior recognition model can be shortened. Specifically, the reason for the long training time of the video behavior recognition model is that the video data is usually large in volume and takes a long time to process. If the video behavior recognition model is trained directly according to the original video data set, the training time of the video behavior recognition model will be too long. Based on this, the video behavior recognition method based on multimodal large model fine-tuning of some embodiments of the present disclosure first obtains a video behavior recognition data set and an initial video behavior recognition model. Among them, each video behavior recognition data in the above-mentioned video behavior recognition data set includes a video frame sequence and a behavior label, and the above-mentioned initial video behavior recognition model includes a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module and a text vector embedding module. Here, the original video data set used to construct the training sample and the initial video behavior recognition model to be trained can be obtained. Then, for each video behavior recognition data in the above video behavior recognition data set, the following steps are performed: determine the key frame image corresponding to the video frame sequence included in the above video behavior recognition data; store the above key frame image to a preset image storage location to generate an image storage path; generate a training sample based on a preset label text template, the above key frame image, the above image storage path and the behavior label included in the above video behavior recognition data. Among them, the above training sample includes a sample image and a sample label data. Thus, a training sample set consisting of a key frame image and a label text can be constructed. Afterwards, based on each generated training sample, the initial video behavior recognition model is trained, and the trained initial video behavior recognition model is determined as a video behavior recognition model. Thus, a video behavior recognition model can be trained. Finally, in response to receiving a behavior recognition request information for the target video data, the above target video data is input into the above video behavior recognition model to obtain video behavior text information, and the above video behavior text information is sent to the user end corresponding to the above behavior recognition request information. Therefore, the video behavior recognition method based on multimodal large model fine-tuning in some embodiments of the present disclosure can greatly reduce the amount of data used to train the initial video behavior recognition model by constructing a training sample set including key frame images and label texts. Thus, the training time of the video behavior recognition model can be shortened so that it can be used to perform behavior recognition on the video data to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flowchart of some embodiments of a video behavior recognition method based on multimodal large model fine-tuning according to the present disclosure;

[0016] Figure 2 is a comparison diagram of the space occupied by video data and image data used to construct a training sample set in some embodiments of the video behavior recognition method based on multimodal large model fine-tuning disclosed in the present invention;

[0017] Figure 3 It is an application scenario diagram of some embodiments of the video behavior recognition method based on multimodal large model fine-tuning according to the present disclosure;

[0018] Figure 4 is a performance evaluation comparison chart of the video behavior recognition model and the multimodal large model according to the present disclosure;

[0019] Figure 5 It is a structural schematic diagram of some embodiments of a video behavior recognition device based on multimodal large model fine-tuning according to the present disclosure;

[0020] Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0023] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0024] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0026] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0027] Figure 1 The flowchart of some embodiments of the video behavior recognition method based on multimodal large model fine-tuning according to the present disclosure is shown. The video behavior recognition method based on multimodal large model fine-tuning 100 includes the following steps:

[0028] Step 101: Obtain a video behavior recognition dataset and an initial video behavior recognition model.

[0029] In some embodiments, the execution subject (e.g., computing device) of the video behavior recognition method based on multimodal large model fine-tuning can obtain the video behavior recognition data set and the initial video behavior recognition model through a wired connection or a wireless connection. Among them, each video behavior recognition data in the above video behavior recognition data set may include a video frame sequence and a behavior label. The above video frame sequence may represent an action video clip. The above behavior label may represent the behavior category corresponding to the action video clip. For example, the behavior category may be but is not limited to one of the following: running, playing a musical instrument, skiing. The above initial video behavior recognition model may include a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module, and a text vector embedding module. The above multimodal large model may be a public model that can support data input and output of multiple modalities (e.g., text, image, video, sound). The above multimodal large model may be a base large model of the initial video behavior recognition model. The above image mapping module may be used to map an image into a high-dimensional vector representation. For example, the above image mapping module may be a pre-trained convolutional neural network. The above image vector embedding module may be used to embed the image vector into the feature space of the multimodal large model. The text mapping module can be used to map text into a high-dimensional vector representation. For example, the text mapping module can be a pre-trained recurrent neural network. The text vector embedding module can be used to embed the text vector into the feature space of the multimodal large model.

[0030] As an example, the video behavior recognition dataset may be the public Kinetics-400 dataset. It should be noted that the wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0031] Step 102: for each video behavior recognition data in the video behavior recognition data set, perform the following steps:

[0032] Step 1021: Determine a key frame image corresponding to a video frame sequence included in the video behavior recognition data.

[0033] In some embodiments, the execution subject may determine the key frame image corresponding to the video frame sequence included in the video behavior recognition data in various ways, wherein the key frame image may be the video frame in the video frame sequence with the largest action change between the previous and next frames.

[0034] It should be noted that the present disclosure takes into account that when solving the problem of "how to determine the key frame images of a video frame sequence", the conventional solution is generally: an algorithm based on image features or a method based on deep learning to extract key frame images from video data, and the above conventional solutions have the following problem three: when the lens motion or video content changes drastically, the algorithm based on image features is difficult to extract relatively accurate key frames. If a deep learning model is used, the key frames extracted are not accurate enough due to the high complexity of the model itself, which further makes the trained video behavior recognition model less accurate, which can easily lead to incorrect recognition of video behaviors. Therefore, in the face of the above technical problem three, combined with the technical advantages of the solution development team itself in the field of key frame extraction, the present disclosure decides to adopt the following solution.

[0035] In some optional implementations of some embodiments, the execution subject may determine the key frame image corresponding to the video frame sequence included in the video behavior recognition data through the following steps:

[0036] In the first step, the video frame sequence included in the video behavior recognition data is processed frame by frame to obtain a set of video frame subsequences, wherein each video frame subsequence may be any three adjacent video frames in the video frame sequence.

[0037] As an example, when the video frame sequence is {frame 1, frame 2, frame 3, frame 4, frame 5}, the above-mentioned video frame subsequence set can be {{frame 1, frame 2, frame 3}, {frame 2, frame 3, frame 4}, {frame 3, frame 4, frame 5}}.

[0038] In the second step, for each video frame subsequence in the above video frame subsequence set, perform the following steps:

[0039] The first sub-step is to determine the inter-frame difference images between the respective video frames in the above-mentioned video frame sub-sequence. The inter-frame difference images may be difference images between an intermediate frame and two adjacent frames. The inter-frame difference images between the respective video frames in the above-mentioned video frame sub-sequence may be determined by the following formula: .

[0040] in, Indicates the frame number of the intermediate frame. Indicates The inter-frame difference images corresponding to the frames. Indicates Frame video frame. Indicates Frame video frame. Indicates Frame video frame.

[0041] The second sub-step is to determine the energy information corresponding to the inter-frame difference image. The energy information can characterize the degree of change of the inter-frame difference image. First, the result of adding the squares of the pixel values ​​in the inter-frame difference image is determined as the image energy value. Then, the inter-frame difference image and the image energy value are determined as the energy information.

[0042] The third step is to select energy information that meets a preset energy condition from the determined energy information as target energy information, wherein the preset energy condition may be energy information in which the image energy value included in each energy information is the maximum value among each image energy value.

[0043] The fourth step is to determine the video frame subsequence corresponding to the target energy information as the target video frame subsequence. The video frame subsequence corresponding to the inter-frame difference image included in the target energy information can be determined as the target video frame subsequence.

[0044] The fifth step is to generate a key frame image based on the target video frame subsequence, wherein the middle frame of the target video frame subsequence can be determined as the key frame image.

[0045] The above-mentioned key frame image generation step and its related contents, as an inventive point of the embodiment of the present disclosure, solve the above-mentioned technical problem three "easy to cause misidentification of video behavior". The reason why conventional solutions are prone to misidentification of video behavior is that when the lens motion or video content changes drastically, it is difficult for the algorithm based on image features to extract relatively accurate key frames. If a deep learning model is used, the key frames extracted are not accurate enough due to the high complexity of the model itself, and the accuracy of the trained video behavior recognition model is further reduced. If this solution solves the above-mentioned problem, the probability of misidentification of video behavior can be reduced. In order to achieve this effect, first, the video frame sequence is grouped into continuous frames. Then, for each group, the inter-frame difference between the middle frame and the adjacent frames is determined. After that, the group with the largest inter-frame difference is selected from each group. Finally, the middle frame in the group with the largest inter-frame difference is used as a key frame image. Thus, the video frame with the most obvious action or the largest action amplitude in the entire video frame sequence can be obtained as a representative key frame image. Further, the model can be trained by the obtained relatively accurate key frame image, thereby making the trained video behavior recognition model more accurate. Thus, the chance of misidentifying video behavior is reduced.

[0046] Step 1022: Store the key frame image to a preset image storage location to generate an image storage path.

[0047] In some embodiments, the execution subject may store the key frame image to a preset image storage location to generate an image storage path. The preset image storage location may be a pre-set address for storing the key frame image. The image storage path may be a storage path of the key frame image.

[0048] As an example, the execution subject may store the key frame image to a preset image storage location, and obtain the image storage path of the key frame image.

[0049] Step 1023: Generate training samples based on the preset label text template, key frame images, image storage path, and behavior labels included in the video behavior recognition data.

[0050] In some embodiments, the execution entity may generate a training sample based on a preset label text template, the key frame image, the image storage path, and the behavior label included in the video behavior recognition data. The training sample may include a sample image and sample label data. The sample image may be an image used for model training. The sample label data may be the annotated true value of the behavior category corresponding to the sample image. The label text template may be a pre-set template in a text format for filling in the image storage path and behavior label.

[0051] In some optional implementations of some embodiments, the execution subject may generate a training sample based on a preset label text template, the key frame image, the image storage path, and the behavior label included in the video behavior recognition data through the following steps:

[0052] The first step is to fill the image storage path and the behavior label included in the video behavior recognition data into a preset label text template to obtain the filled label text template as sample label data.

[0053] In the second step, the key frame image is used as a sample image, and the sample image and the sample label data are determined as training samples.

[0054] As an example, the above label text template can be {"message": [{"text content":" What is this behavior?", "role": "questioner"}, {"text content": this behavior is [activity_class], "role": "answerer"}], "image": [image_path]}. Among them, Indicates the data in image form for training. [activity_class] indicates the behavior label included in the video behavior recognition data to be filled. [image_path] indicates the image storage path to be filled.

[0055] To continue, please refer to Figure 2 , Figure 2 The figure shows the comparison of the space occupied by video data and image data used to construct the training sample set in some embodiments of the video behavior recognition method based on multimodal large model fine-tuning according to the present disclosure. Obviously, after the above step 102, the overall data amount used to train the initial video behavior recognition model can be greatly reduced.

[0056] Step 103: training the initial video behavior recognition model based on the generated training samples, and determining the trained initial video behavior recognition model as the video behavior recognition model.

[0057] In some embodiments, the execution entity may train the initial video behavior recognition model based on the generated training samples through various training methods, and determine the trained initial video behavior recognition model as the video behavior recognition model.

[0058] In some optional implementations of some embodiments, the execution subject may perform the following training steps based on the generated training samples to train the initial video behavior recognition model:

[0059] In the first step, the sample images included in each of the above-mentioned training samples are input into the image mapping module in the initial video behavior recognition model to output image modality feature vectors to obtain each image modality feature vector. Among them, each of the above-mentioned image modality feature vectors can correspond one-to-one to the training samples in the above-mentioned training samples. Each of the above-mentioned image modality feature vectors can be a high-dimensional vector representation of the sample images included in the training samples. It should be noted that the above-mentioned image mapping module is a pre-trained module, and the parameters are kept frozen during the training process of the initial video behavior recognition model.

[0060] In the second step, each of the above-mentioned image modality feature vectors is input into the image vector embedding module in the initial video behavior recognition model to output image features and obtain each image feature. Among them, each of the above-mentioned image features can correspond one-to-one to the image modality feature vector in each image modality feature vector. The image features in the above-mentioned image features can be the result of embedding the corresponding image modality feature vector into the feature space of the multimodal large model.

[0061] In some optional implementations of some embodiments, the above-mentioned image vector embedding module may include an attention layer and a linear layer. Among them, the above-mentioned attention layer can be used to output an attention weighted vector according to the input vector through an attention mechanism. Among them, the above-mentioned attention weighted vector can represent the degree of attention to different areas or features of the image. The above-mentioned linear layer can be used to convert the input vector from one feature space to another feature space. The above-mentioned execution subject can input each of the above-mentioned image modality feature vectors into the image vector embedding module in the initial video behavior recognition model to output image features and obtain each image feature through the following steps:

[0062] Step 1: for each of the above-mentioned image modality feature vectors, perform the following steps:

[0063] Sub-step 1: Determine the attention weighted vector corresponding to the above-mentioned image modality feature vector through the attention layer included in the above-mentioned image vector embedding module based on the preset learnable weight matrix group. The above-mentioned learnable weight matrix group may include a query weight matrix, a key weight matrix and a value weight matrix. The attention weighted vector can be generated by the following formula group: .

[0064] in, Represents an image. represents the modal eigenvector. Represents the query vector. Represents a key vector. represents a value vector, and , , . represents the learnable query weight matrix. represents a learnable key weight matrix. represents the learnable value weight matrix, and , , . Represents the dot product between the query and the key. Indicates the dimension of the key vector. Used to scale the dot product to prevent the vanishing gradient problem. Represents the activation function. represents the attention weight. represents the attention weight vector.

[0065] Sub-step two, through the linear layer included in the above-mentioned image vector embedding module, based on the preset learnable linear layer weight matrix and the learnable bias vector, the attention weighted vector corresponding to the above-mentioned image modality feature vector is embedded into the preset vector space, and the embedded attention weighted vector is obtained as the image feature. Among them, the above-mentioned preset vector space can be the vector space of the above-mentioned multimodal large model. The above-mentioned learnable linear layer weight matrix can be the weight matrix of the learnable linear layer. Each element in the above-mentioned learnable linear layer weight matrix can belong to the above-mentioned preset vector space. The above-mentioned learnable bias vector can be a learnable bias vector in the above-mentioned preset vector space. The linear layer included in the above-mentioned image vector embedding module can embed the attention weighted vector corresponding to the above-mentioned image modality feature vector into the preset vector space based on the preset learnable linear layer weight matrix and the learnable bias vector through the following formula to obtain the image feature: .

[0066] in, Represents image features. represents the learnable linear layer weight matrix, and . represents the learnable bias vector, and . Represents the above preset vector space.

[0067] The third step is to perform mask processing on the sample label data included in each of the above training samples to generate mask label texts to obtain each mask label text. The mask label text in each of the above mask label texts may be sample label data after masking characters that are not directly related to the behavior recognition task. The sample label data included in each of the above training samples may be masked by masking technology to generate mask label texts to obtain each mask label text.

[0068] The fourth step is to input each of the above-mentioned masked label texts into the text mapping module in the initial video behavior recognition model to output a text modality feature vector to obtain each text modality feature vector. Among them, each of the above-mentioned text modality feature vectors can correspond one-to-one to a training sample in the above-mentioned training samples. Each of the above-mentioned text modality feature vectors can be a high-dimensional vector representation of the sample label data included in the training sample. It should be noted that the above-mentioned text mapping module is a pre-trained module, and the parameters are kept frozen during the training process of the initial video behavior recognition model.

[0069] In the fifth step, each of the above-mentioned text modal feature vectors is input into the text vector embedding module in the initial video behavior recognition model to output text features and obtain each text feature. Each of the above-mentioned text features can correspond one-to-one to the text modal feature vector in each text modal feature vector. The text features in each of the above-mentioned text features can be the result of embedding the corresponding text modal feature vector into the feature space of the multimodal large model.

[0070] In some optional implementations of some embodiments, the execution subject may input each of the above-mentioned text modality feature vectors into a text vector embedding module in the initial video behavior recognition model, and perform the following steps to generate each text feature:

[0071] For each of the above-mentioned text modal feature vectors, the above-mentioned text modal feature vector is embedded into the above-mentioned preset vector space through the above-mentioned text vector embedding module according to the preset low-rank adaptive fine-tuning method, and the embedded text modal feature vector is obtained as the text feature.

[0072] As an example, the above execution body first represents the incremental parameter matrix of full parameter fine-tuning as a low-rank approximation of two matrices with smaller parameter amounts for the pre-trained weight matrix, and the formula is: .

[0073] in, represents the pre-trained weight matrix, and . Represents the incremental parameter matrix for all parameter fine-tuning. represents a low-rank matrix, and . represents a low-rank matrix, and .

[0074] Then, for each of the above-mentioned text modality feature vectors, the above-mentioned text modality feature vector is embedded into the above-mentioned preset vector space through the following formula corresponding to the linear layer, and the embedded text modality feature vector is obtained as the text feature: .

[0075] in, Represents text. Represents the text modality feature vector. Represents text features. represents a learnable weight matrix, and . represents the bias term, and .

[0076] Step 6: Generate a training loss value based on the above image features and the above text features. The above training loss value can be used to characterize the difference between different modal features. The training loss value can be generated based on the above image features and the above text features in various ways.

[0077] Continuing, in the process of generating training loss values, the present disclosure is also accompanied by the following problem: "How to construct a loss function according to the differences between different modal features to achieve modal feature alignment through back propagation. The conventional solution is generally: construct a modal alignment loss function through the similarity between image features and text features to achieve modal feature alignment through back propagation. However, the above conventional solution still has the following problem four: due to ignoring the modal alignment between video features and image features and text features in the video recognition task, the trained video behavior recognition model has low accuracy, which easily leads to incorrect recognition of video behaviors. Therefore, in the face of the above technical problem four, combined with the technical advantages of the solution development team itself in the field of deep learning, the present disclosure decided to adopt the following solution.

[0078] In some optional implementations of some embodiments, the execution subject may generate a training loss value based on the above-mentioned image features and the above-mentioned text features through the following steps:

[0079] Step 1: Determine the modal loss value between each of the above image features and the corresponding target text feature through a custom modal alignment loss function to obtain each modal loss value. The target text feature can be a text feature of the above text features that corresponds to the same video behavior recognition data as the image feature. The above modal alignment loss function can be: .

[0080] in, Represents the modal loss value. Represents the total number of sample label data. Indicates the sequence number of the sample label data. Indicates the sequence number of the sample image. Indicates The image features of the sample images. Indicates the sequence number of the sample label data. Indicates The sample images correspond to the first The text features corresponding to the sample label data. Represents the similarity function. Indicates The text features corresponding to the sample label data.

[0081] Step 2, through the customized intra-class difference loss function and inter-class difference loss function, based on the pre-generated video features, the above-mentioned image features and the above-mentioned text features, generate intra-class difference loss values ​​and inter-class difference loss values. Among them, each of the above-mentioned video features can correspond one-to-one to the video frame sequence in the above-mentioned video behavior recognition dataset. Each of the above-mentioned video features can be obtained by extracting features from the video frame sequence through the above-mentioned multimodal large model. The above-mentioned intra-class difference loss value can characterize the difference between different modal data within the behavior category. The above-mentioned inter-class difference loss value can characterize the difference between modal data between different behavior categories. The above-mentioned intra-class difference loss function can be: .

[0082] in, Indicates intra-class differences. Represents the intra-class difference loss value. Represents video features. Indicates the sequence number of the training sample. Represents the total number of training samples. A serial number indicating the behavior category. Indicates The total number of training samples for each behavior category. Represents the total number of behavior categories. Indicates The video features corresponding to the training samples. Indicates The image features corresponding to the training samples. Indicates The text features corresponding to the training samples. Represents standard deviation. Indicates The standard deviation between the image features and video features corresponding to the training samples. Indicates The standard deviation between the image features and text features corresponding to the training samples. Indicates The standard deviation between the video features and text features corresponding to the training samples.

[0083] Continuing, the above inter-class difference loss function can be: .

[0084] in, Represents the difference between classes. and Indicates different categories of behavior. Indicates Class behavior category and The two-class difference loss between class behavior categories. Indicates Class behavior category and The inter-class difference loss between class behaviors with respect to image features. Indicates Class behavior category and The inter-class difference loss between class behavior categories with respect to text features. Indicates Class behavior category and Inter-class difference loss between class behavior categories with respect to video features. Indicates the type of modality feature, which can be taken from image, text, and video. Indicates Class behavior category and Class behavior categories Inter-class discrepancy loss for type modality features. Indicates Class behavior category and Class behavior categories Inter-class differences in type modal features. Indicates About the behavior category Modal features of type modal features. Indicates About the behavior category Modal features of type modal features.

[0085] As an example, the execution subject may first generate the intra-class difference loss value corresponding to each behavior category through the intra-class difference loss function. Then, for any two behavior categories in each behavior category, the inter-class difference loss between the two behavior categories may be determined through the inter-class difference loss function. Finally, the sum of the obtained inter-class difference losses is determined as the inter-class difference loss value.

[0086] Step 3: Generate a training loss value based on the above-mentioned modal loss values, the above-mentioned intra-class difference loss value and the above-mentioned inter-class difference loss value. First, the sum of the above-mentioned modal loss values ​​is determined as the modal alignment loss value. Then, the above-mentioned modal alignment loss value, the above-mentioned intra-class difference loss value and the above-mentioned inter-class difference loss value are weighted and summed to obtain the training loss value.

[0087] In step 7, in response to determining that the training loss value is less than the preset loss threshold, the trained initial video behavior recognition model is determined as the video behavior recognition model. The preset loss threshold may be an upper limit of the pre-set training loss value.

[0088] Optionally, the execution subject may adjust the learnable network parameters of the initial video behavior recognition model in response to determining that the training loss value is greater than or equal to the preset loss threshold, and use the adjusted initial video behavior recognition model as the initial video behavior recognition model, and perform the training step again based on the above training samples. The learnable network parameters in the corresponding initial video behavior recognition model may be optimized by a gradient descent method.

[0089] The above training loss value generation step and its related content, as an inventive point of an embodiment of the present disclosure, solve the above technical problem four "easy to lead to wrong recognition of video behavior". The reason why conventional solutions are prone to wrong recognition of video behavior is that the accuracy of the trained video behavior recognition model is low due to the neglect of the modal alignment between video features, image features and text features in the video recognition task. If this solution solves the above problem, the effect of improving the probability of correctly identifying video behavior can be achieved. In order to achieve this effect, first, the modal loss between image features and text features can be determined. Then, the modal loss between each modal feature in the same video behavior category can be determined according to the above-mentioned customized intra-class difference loss function. After that, the modal loss of each modal feature between different video behavior categories can be determined according to the above-mentioned customized inter-class difference loss function. Finally, the training loss value is determined based on all the modal losses obtained. In this way, the video, image and text can be aligned to improve the accuracy of the video behavior recognition model. Thus, the probability of correctly identifying video behavior can be improved.

[0090] Optionally, the multimodal large model included in the initial video behavior recognition model freezes parameters during the training process of the initial video behavior recognition model.

[0091] The above-mentioned initial video behavior recognition model training steps and related contents, as an inventive point of an embodiment of the present disclosure, solve the above-mentioned technical problem 2 "it is difficult to perform behavior recognition on the video data to be detected in a timely manner". The reason why conventional solutions are difficult to perform behavior recognition on the video data to be detected in a timely manner is that if the multimodal video behavior recognition model is fully parameter trained, a large amount of computing resources and time will be consumed, resulting in excessive occupation of computing resources and a long model training time. If this solution solves the above-mentioned problem, the effect of timely performing behavior recognition on the video data to be detected can be achieved. In order to achieve this effect, firstly, on the basis of the multimodal large model with frozen parameters, additional modules such as image mapping module, image vector embedding module, text mapping module and text vector embedding module can be added. Then, the learnable parameters in the image vector embedding module and the text vector embedding module can be optimized and adjusted by the model fine-tuning method. Finally, in response to determining that the above-mentioned training loss value is less than the above-mentioned preset loss threshold, the trained initial video behavior recognition model is determined as the video behavior recognition model. Therefore, by fine-tuning the parameters of the initial video behavior recognition model including the multimodal large model, the computing resources and time occupied can be reduced, thereby shortening the model training time for behavior recognition of the video data to be detected.

[0092] Step 104, in response to receiving the behavior recognition request information for the target video data, input the target video data into the video behavior recognition model to obtain the video behavior text information, and send the video behavior text information to the user end corresponding to the behavior recognition request information.

[0093] In some embodiments, the above-mentioned execution subject may, in response to receiving a behavior recognition request information for the target video data, input the above-mentioned target video data into the above-mentioned video behavior recognition model to obtain video behavior text information, and send the above-mentioned video behavior text information to the user end corresponding to the behavior recognition request information. Among them, the above-mentioned target video data may be a video frame sequence to be subjected to behavior recognition. The above-mentioned behavior recognition request information may be a text input by a user requesting behavior recognition of the above-mentioned target video data. For example, the above-mentioned behavior recognition request information may be "What is the person in the video doing?". The above-mentioned video behavior text information may be information on the behavior category corresponding to the action behavior in the above-mentioned target video data.

[0094] As an example, the execution subject may first fill the storage path of the video into the label text template. Then, through the video behavior recognition model, the video features of the target video data are learned, and the learned video features are matched with the pre-learned image features to obtain the image features with the highest matching degree as the target image features. Finally, the behavior label corresponding to the target image feature is determined as the video behavior text information.

[0095] In practice, before inputting the target video data into the video behavior recognition model to obtain the video behavior text information, the execution subject may also use the transformer-based intention recognition model to identify the behavior recognition request information and confirm that the behavior recognition request information corresponds to the behavior recognition task. After that, the video behavior recognition model is instructed to generate the video behavior text information according to the target video data.

[0096] Optionally, the user terminal may display or play the received video behavior text information.

[0097] As an example, continue to refer to Figure 3 and Figure 4 . Figure 3 The following diagrams show some application scenarios of the video behavior recognition method based on multimodal large model fine-tuning according to the present disclosure. Figure 3 The video on the left is the target video data that needs to be used for behavior recognition, and the right side provides the answers to the behavior recognition questions of the large base model and the video behavior recognition model of the present invention respectively. Obviously, compared with the large base model, the behavior category output by the video behavior recognition model of the present invention is more accurate.

[0098] in addition, Figure 4 The performance evaluation comparison diagram of the video behavior recognition model according to the present disclosure and the multimodal large model is shown. Figure 4 It is a display of the results of the performance test comparison experiment of the multimodal large model (base large model) and the video behavior recognition model disclosed in this disclosure. In the performance test comparison experiment, the test data uses K400 video data, including video training sets and video test sets, and the indicators used to answer the evaluation include indicator 1, indicator 2, indicator 3, and indicator 4. The above indicators 1, indicator 2, indicator 3, and indicator 4 are BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-1, respectively, and there is also an accuracy rate for evaluating the accuracy of behavior recognition. The experimental results show that the recognition rate of the base large model for unknown video behavior samples is extremely low, not exceeding 11%, and it is basically unable to recognize unknown behaviors. This is because the large model has not been trained with video behavior recognition data. After the video behavior recognition model disclosed in this disclosure is fine-tuned with the lightweight K400 image training set, all indicators show a significant improvement.

[0099] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the video behavior recognition method based on multimodal large model fine-tuning of some embodiments of the present disclosure, the training time of the video behavior recognition model can be shortened. Specifically, the reason for the long training time of the video behavior recognition model is that the video data is usually large in volume and takes a long time to process. If the video behavior recognition model is trained directly according to the original video data set, the training time of the video behavior recognition model will be too long. Based on this, the video behavior recognition method based on multimodal large model fine-tuning of some embodiments of the present disclosure first obtains a video behavior recognition data set and an initial video behavior recognition model. Among them, each video behavior recognition data in the above-mentioned video behavior recognition data set includes a video frame sequence and a behavior label, and the above-mentioned initial video behavior recognition model includes a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module and a text vector embedding module. Here, the original video data set used to construct the training sample and the initial video behavior recognition model to be trained can be obtained. Then, for each video behavior recognition data in the above video behavior recognition data set, the following steps are performed: determine the key frame image corresponding to the video frame sequence included in the above video behavior recognition data; store the above key frame image to a preset image storage location to generate an image storage path; generate a training sample based on a preset label text template, the above key frame image, the above image storage path and the behavior label included in the above video behavior recognition data. Among them, the above training sample includes a sample image and a sample label data. Thus, a training sample set consisting of a key frame image and a label text can be constructed. Afterwards, based on each generated training sample, the initial video behavior recognition model is trained, and the trained initial video behavior recognition model is determined as a video behavior recognition model. Thus, a video behavior recognition model can be trained. Finally, in response to receiving a behavior recognition request information for the target video data, the above target video data is input into the above video behavior recognition model to obtain video behavior text information, and the above video behavior text information is sent to the user end corresponding to the above behavior recognition request information. Therefore, the video behavior recognition method based on multimodal large model fine-tuning in some embodiments of the present disclosure can greatly reduce the amount of data used to train the initial video behavior recognition model by constructing a training sample set including key frame images and label texts. Thus, the training time of the video behavior recognition model can be shortened so that it can be used to perform behavior recognition on the video data to be detected.

[0100] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a video behavior recognition device based on multimodal large model fine-tuning. These device embodiments are similar to Figure 1Corresponding to the method embodiments shown, the video behavior recognition device 500 based on multimodal large model fine-tuning can be specifically applied to various electronic devices.

[0101] like Figure 5 As shown, some embodiments of the video behavior recognition device 500 based on multimodal large model fine-tuning include: an acquisition unit 501, an execution unit 502, a training and determination unit 503 and an input and sending unit 504. Among them, the acquisition unit 501 is configured to acquire a video behavior recognition data set and an initial video behavior recognition model, wherein each video behavior recognition data in the above video behavior recognition data set includes a video frame sequence and a behavior label, and the above initial video behavior recognition model includes a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module and a text vector embedding module; the execution unit 502 is configured to perform the following steps for each video behavior recognition data in the above video behavior recognition data set: determine the key frame image corresponding to the video frame sequence included in the above video behavior recognition data; store the above key frame image to a preset image storage location to generate an image storage path; based on a preset label text template, The key frame images, the image storage paths and the behavior labels included in the video behavior recognition data are used to generate training samples, wherein the training samples include sample images and sample label data; the training and determination unit 503 is configured to train the initial video behavior recognition model based on the generated training samples, and to determine the trained initial video behavior recognition model as the video behavior recognition model; the input and sending unit 504 is configured to respond to receiving behavior recognition request information for target video data, input the target video data into the video behavior recognition model, obtain video behavior text information, and send the video behavior text information to the user end corresponding to the behavior recognition request information.

[0102] It can be understood that the units recorded in the video behavior recognition device 500 based on multimodal large model fine-tuning are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the video behavior recognition device 500 based on multimodal large model fine-tuning and the units contained therein, and will not be described in detail here.

[0103] Further references Figure 6 , which shows a structural schematic diagram of an electronic device 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0104] like Figure 6As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0106] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0107] It should be noted that the computer-readable medium mentioned above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0108] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0109] The above-mentioned computer-readable medium may be included in the above-mentioned device; or it may exist independently and not be assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, as an example, the electronic device may execute: obtaining a video behavior recognition data set and an initial video behavior recognition model, wherein each video behavior recognition data in the above-mentioned video behavior recognition data set includes a video frame sequence and a behavior label, and the above-mentioned initial video behavior recognition model includes a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module and a text vector embedding module; for each video behavior recognition data in the above-mentioned video behavior recognition data set, perform the following steps: determine the key frame image corresponding to the video frame sequence included in the above-mentioned video behavior recognition data; store the above-mentioned key frame image to Preset an image storage location to generate an image storage path; generate training samples based on a preset label text template, the key frame image, the image storage path and the behavior label included in the video behavior recognition data, wherein the training samples include sample images and sample label data; train an initial video behavior recognition model based on each generated training sample, and determine the trained initial video behavior recognition model as a video behavior recognition model; in response to receiving behavior recognition request information for target video data, input the target video data into the video behavior recognition model to obtain video behavior text information, and send the video behavior text information to a user terminal corresponding to the behavior recognition request information.

[0110] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising: an acquisition unit, an execution unit, a training and determination unit, and an input and sending unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring a video behavior recognition data set and an initial video behavior recognition model".

[0113] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0114] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A video behavior recognition method based on fine-tuning of a multimodal large model, comprising: Obtain a video behavior recognition data set and an initial video behavior recognition model, wherein each video behavior recognition data in the video behavior recognition data set includes a video frame sequence and a behavior label, and the initial video behavior recognition model includes a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module, and a text vector embedding module; For each video behavior recognition data in the video behavior recognition data set, the following steps are performed: Determine a key frame image corresponding to the video frame sequence included in the video behavior recognition data, wherein the key frame image is a video frame in the video frame sequence having the largest action change between the previous and next adjacent frames, including: The first step is to perform frame-by-frame grouping processing on the video frame sequence included in the video behavior recognition data to obtain a video frame subsequence set; In the second step, for each video frame subsequence in the video frame subsequence set, the following steps are performed: In the first sub-step, an inter-frame difference image between each video frame in the video frame sub-sequence is determined by the following formula: D t =|frame t-1 -2frame t +frame t+1 |, Where t represents the frame number of the intermediate frame, D t Represents the inter-frame difference image corresponding to the tth frame, frame t-1 Indicates the t-1th frame of the video, frame t Indicates the t-th frame of the video, frame t+1 Represents the t+1th video frame; The second sub-step is to determine energy information corresponding to the inter-frame difference image, wherein the energy information represents the degree of change of the inter-frame difference image, wherein the result of adding the squares of each pixel value in the inter-frame difference image is determined as the image energy value, and the inter-frame difference image and the image energy value are determined as the energy information, The third step is to select energy information that meets a preset energy condition from the determined energy information as target energy information, wherein the preset energy condition is energy information in which the image energy value included in each energy information is the maximum value among each image energy value; The fourth step is to determine the video frame subsequence corresponding to the target energy information as the target video frame subsequence, wherein the video frame subsequence corresponding to the inter-frame difference image included in the target energy information is determined as the target video frame subsequence; Step 5: generating a key frame image based on the target video frame subsequence, wherein an intermediate frame of the target video frame subsequence is determined as the key frame image; Storing the key frame image in a preset image storage location to generate an image storage path; Generate a training sample based on a preset label text template, the key frame image, the image storage path, and the behavior label included in the video behavior recognition data, wherein the training sample includes a sample image and sample label data; Based on the generated training samples, an initial video behavior recognition model is trained, and the trained initial video behavior recognition model is determined as a video behavior recognition model; in response to receiving behavior recognition request information for target video data, the target video data is input into the video behavior recognition model to obtain video behavior text information, and the video behavior text information is sent to a user terminal corresponding to the behavior recognition request information.

2. The method according to claim 1, wherein: The generating of training samples based on the preset label text template, the key frame image, the image storage path and the behavior label included in the video behavior recognition data includes: Filling the image storage path and the behavior label included in the video behavior recognition data into a preset label text template to obtain the filled label text template as sample label data; The key frame image is used as a sample image, and the sample image and the sample label data are determined as training samples.

3. The method according to claim 1, wherein: The initial video behavior recognition model is trained based on the generated training samples, including: Based on the generated training samples, the following training steps are performed: Inputting the sample image included in each training sample in the training samples into the image mapping module in the initial video behavior recognition model to output the image modality feature vector, thereby obtaining each image modality feature vector; Inputting each of the image modality feature vectors into an image vector embedding module in an initial video behavior recognition model to output image features, thereby obtaining each image feature; Performing mask processing on the sample label data included in each of the training samples to generate mask label text, thereby obtaining each mask label text; Inputting each mask label text in the respective mask label texts into a text mapping module in the initial video behavior recognition model to output a text modality feature vector, thereby obtaining respective text modality feature vectors; Inputting each of the text modality feature vectors into a text vector embedding module in the initial video behavior recognition model to output text features, thereby obtaining each text feature; Generate a training loss value based on the respective image features and the respective text features; In response to determining that the training loss value is less than a preset loss threshold, the trained initial video behavior recognition model is determined as the video behavior recognition model.

4. The method according to claim 3, wherein: The method further comprises: In response to determining that the training loss value is greater than or equal to the preset loss threshold, adjusting the learnable network parameters of the initial video behavior recognition model, and using the adjusted initial video behavior recognition model as the initial video behavior recognition model, and performing the training step again based on the respective training samples.

5. The method according to claim 3, wherein: The image vector embedding module includes an attention layer and a linear layer; and the image vector embedding module inputting each of the image modality feature vectors into the initial video behavior recognition model to output image features includes: For each of the image modality feature vectors, the following steps are performed: Determining, by an attention layer included in the image vector embedding module, an attention weighted vector corresponding to the image modality feature vector based on a preset learnable weight matrix group, wherein the learnable weight matrix group includes a query weight matrix, a key weight matrix, and a value weight matrix; Through the linear layer included in the image vector embedding module, based on the preset learnable linear layer weight matrix and the learnable bias vector, the attention weighted vector corresponding to the image modality feature vector is embedded into the preset vector space to obtain the embedded attention weighted vector as the image feature, wherein the preset vector space is the vector space of the multimodal large model, each element in the learnable linear layer weight matrix belongs to the preset vector space, and the learnable bias vector is a vector in the preset vector space.

6. The method according to claim 5, wherein: The step of inputting each of the text modality feature vectors into a text vector embedding module in the initial video behavior recognition model to output text features, thereby obtaining each text feature, including: For each of the text modal feature vectors, the text modal feature vector is embedded into the preset vector space through the text vector embedding module according to a preset low-rank adaptive fine-tuning method, and the embedded text modal feature vector is obtained as a text feature.

7. The method according to any one of claims 1 to 6, wherein: The multimodal large model included in the initial video behavior recognition model freezes parameters during the training process of the initial video behavior recognition model.

8. A video behavior recognition device based on fine-tuning of a multimodal large model, comprising: an acquisition unit configured to acquire a video behavior recognition data set and an initial video behavior recognition model, wherein each video behavior recognition data in the video behavior recognition data set includes a video frame sequence and a behavior label, and the initial video behavior recognition model includes a multimodal large model, an image mapping module, an image vector embedding module, a text mapping module, and a text vector embedding module; The execution unit is configured to execute the following steps for each video behavior recognition data in the video behavior recognition data set: Determine a key frame image corresponding to the video frame sequence included in the video behavior recognition data, wherein the key frame image is a video frame in the video frame sequence having the largest action change between the previous and next adjacent frames, including: The first step is to perform frame-by-frame grouping processing on the video frame sequence included in the video behavior recognition data to obtain a video frame subsequence set; In the second step, for each video frame subsequence in the video frame subsequence set, the following steps are performed: In the first sub-step, an inter-frame difference image between each video frame in the video frame sub-sequence is determined by the following formula: D t =|frame t-1 -2frame t +frame t+1 |, Where t represents the frame number of the intermediate frame, D t Represents the inter-frame difference image corresponding to the tth frame, frame t-1 Indicates the t-1th frame of the video, frame t Indicates the t-th frame of the video, frame t+1 Represents the t+1th video frame; The second sub-step is to determine energy information corresponding to the inter-frame difference image, wherein the energy information represents the degree of change of the inter-frame difference image, wherein the result of adding the squares of each pixel value in the inter-frame difference image is determined as the image energy value, and the inter-frame difference image and the image energy value are determined as the energy information, The third step is to select energy information that meets a preset energy condition from the determined energy information as target energy information, wherein the preset energy condition is energy information in which the image energy value included in each energy information is the maximum value among each image energy value; The fourth step is to determine the video frame subsequence corresponding to the target energy information as the target video frame subsequence, wherein the video frame subsequence corresponding to the inter-frame difference image included in the target energy information is determined as the target video frame subsequence; Step 5: generating a key frame image based on the target video frame subsequence, wherein an intermediate frame of the target video frame subsequence is determined as the key frame image; Storing the key frame image in a preset image storage location to generate an image storage path; Generate a training sample based on a preset label text template, the key frame image, the image storage path, and the behavior label included in the video behavior recognition data, wherein the training sample includes a sample image and sample label data; A training and determining unit is configured to train the initial video behavior recognition model based on the generated training samples, and determine the trained initial video behavior recognition model as the video behavior recognition model; The input and sending unit is configured to, in response to receiving behavior recognition request information for target video data, input the target video data into the video behavior recognition model to obtain video behavior text information, and send the video behavior text information to the user end corresponding to the behavior recognition request information.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images