Micro-motion recognition method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a micro-motion recognition method, device, electronic device and storage medium, which relate to the field of computer vision technology. The method includes: obtaining a video to be recognized containing a specified object; inputting each video frame in the video to be recognized into a visual feature extraction network in a pre-trained micro-motion recognition model to obtain a first video feature to be recognized; calculating the adjustment amount corresponding to the first feature vector to be recognized according to the position sequence of each first feature vector to be recognized in the first video feature to be recognized; using each adjustment amount, adjusting the corresponding first feature vector to be recognized in the first video feature to be recognized to obtain a second video feature to be recognized; based on the matching result of the second video feature to be recognized and the text feature of the preset prompt word, determining whether the specified object in the video to be recognized has a micro-motion represented by the preset prompt word. In this way, micro-motion recognition can be achieved and the accuracy of micro-motion recognition can be guaranteed.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a micro-motion recognition method, device, electronic device and storage medium. Background Art

[0002] With the development of computer vision technology, the demand for human motion recognition is becoming more and more widespread, especially the recognition of human micro-motions has become a challenging task. Human micro-motions refer to subtle body movements or facial expressions of the human body. For example, in a driving scenario, it is necessary to identify whether the driver has micro-motions such as making a phone call, yawning, closing eyes, etc.; for another example, in a smart device control scenario, it is necessary to identify micro-motions such as gestures made by the user.

[0003] However, due to the small amplitude, fast change, and individual differences of micro-motions, it is difficult to identify micro-motions. Therefore, how to achieve micro-motion recognition and ensure its accuracy has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a micro-motion recognition method, device, electronic device and storage medium to achieve micro-motion recognition and ensure the accuracy of micro-motion recognition. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a micro-motion recognition method is first provided, the method comprising:

[0006] Get the video to be identified that contains the specified object;

[0007] Input each video frame in the video to be identified into a visual feature extraction network in a pre-trained micro-motion recognition model to obtain a first video feature to be identified; wherein the first video feature to be identified includes a plurality of first feature vectors to be identified, the plurality of first feature vectors to be identified correspond one-to-one to the video frames included in the video to be identified, and the position sequence of each first feature vector to be identified in the first video feature to be identified is consistent with the time sequence of the corresponding video frame in the video to be identified;

[0008] Calculating the adjustment amount corresponding to each first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified;

[0009] Using each adjustment amount, adjusting the first feature vector to be identified corresponding to the first video feature to be identified, to obtain a second video feature to be identified;

[0010] Based on the matching result of the second to-be-recognized video feature and the text feature of the preset prompt word, determining whether the specified object in the to-be-recognized video has a micro-motion represented by the preset prompt word;

[0011] Among them, the text features of the preset prompt words are obtained by extracting features of the preset prompt words based on the text feature extraction network in the micro-motion recognition model; the micro-motion recognition model is trained based on the first sample video and the text labels corresponding to the first sample video, and the text labels corresponding to each first sample video represent the micro-motions of the specified object in the first sample video.

[0012] In some embodiments, calculating the adjustment amount corresponding to each first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified includes:

[0013] According to the position sequence of each first feature vector to be identified in the first video feature to be identified, the adjustment amount corresponding to the first feature vector to be identified is calculated according to the following formula:

[0014] ;

[0015] The value range of i is between 1 and ; Indicates An adjustment amount corresponding to a first feature vector to be identified; represents the total number of video frames contained in the video to be identified; d represents the dimension of the first feature vector to be identified.

[0016] In some embodiments, the micro-motion recognition model further comprises a sequence network;

[0017] The step of adjusting the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount to obtain the second video feature to be identified includes:

[0018] Using each adjustment amount, adjusting the first feature vector to be identified corresponding to the first video feature to be identified, to obtain an adjusted first video feature to be identified;

[0019] Performing convolution processing on the adjusted first video feature to be identified to obtain multiple feature maps with the same dimensions;

[0020] For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain the second feature vector to be identified corresponding to the position;

[0021] The obtained second feature vectors to be identified are input into the sequence network, and the feature vectors output by the sequence network at multiple moments are combined to obtain the second video features to be identified.

[0022] In some embodiments, the sequence network comprises a hidden layer and an output layer;

[0023] The step of inputting each of the obtained second feature vectors to be identified into the sequence network, and combining the feature vectors output by the sequence network at multiple moments to obtain the second video features to be identified includes:

[0024] Using the weight of the sequence network at the current moment, a weighted sum of the second feature vectors to be identified is calculated as the weighted feature vector at the current moment; wherein the weight of the sequence network at the current moment is obtained based on the second feature vectors to be identified and the feature vector output by the hidden layer at the previous moment;

[0025] Inputting the feature vector output by the output layer at the last moment, the weighted feature vector at the current moment, and the feature vector output by the hidden layer at the last moment into the hidden layer, to obtain the feature vector output by the output layer at the current moment, and the feature vector output by the hidden layer at the current moment;

[0026] Returning to the step of using the weights of the sequence network at the current moment to calculate the weighted sum of the second feature vectors to be identified as the weighted feature vector at the current moment, until a specified number of feature vectors output by the output layer are obtained;

[0027] The specified number of feature vectors output by the output layer are combined to obtain the second video features to be identified.

[0028] In some embodiments, the preset prompt words include preset positive prompt words, and a positive prompt word indicates a positive micro-motion existing in a specified object when a specified event is triggered;

[0029] After determining whether the specified object in the video to be identified has a micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word, the method further includes:

[0030] If the specified object in the video to be identified has the positive micro-motion, then it is determined that the specified object in the video to be identified triggers the specified event.

[0031] In some embodiments, the preset prompt word includes a preset negative prompt word, and the negative prompt word represents a negative example micro-motion other than the positive example micro-motion;

[0032] If the specified object in the video to be identified has the positive micro-motion, determining that the specified object in the video to be identified triggers the specified event includes:

[0033] If the specified object in the video to be identified has the positive micro-motion and does not have the negative micro-motion, it is determined that the specified object in the video to be identified triggers the specified event.

[0034] In some embodiments, the preset prompt word is obtained based on the following steps:

[0035] Obtain each candidate prompt word and a second sample video; wherein the candidate prompt word includes: a plurality of candidate positive prompt words and a plurality of candidate negative prompt words; the second sample video includes a plurality of second positive sample videos and a plurality of second negative sample videos, a specified object in any second positive sample video triggers the specified event, and a specified object in any second negative sample video does not trigger the specified event;

[0036] Input each candidate prompt word into a text feature extraction network in a pre-trained multimodal large model to obtain text features of each candidate prompt word;

[0037] Inputting each second sample video into a visual feature extraction network in a pre-trained multimodal large model to obtain video features of each second sample video;

[0038] For each second sample video, calculating the similarity between the text features of each candidate prompt word and the video features of the second sample video;

[0039] Based on the obtained similarity, a prompt word that matches the second positive sample video and does not match the second negative sample video is determined from each candidate positive prompt word as a preset positive prompt word; and a prompt word that does not match the second positive sample video and matches the second negative sample video is determined from each candidate negative prompt word as a preset negative prompt word.

[0040] In some embodiments, the text feature extraction network in the micro-motion recognition model is a text feature extraction network in a pre-trained multimodal large model.

[0041] In some embodiments, the training process of the multimodal large model includes the following steps:

[0042] Obtaining a third sample video and a text label corresponding to the third sample video; wherein the text label corresponding to each third sample video indicates a micro-motion of a specified object in the third sample video;

[0043] Input each third sample video into the visual feature extraction network in the multimodal large model of the initial structure to obtain the video features of the third sample video;

[0044] Inputting the text label corresponding to each third sample video into the text feature extraction network in the multimodal large model of the initial structure to obtain the text feature of the text label corresponding to the third sample video;

[0045] Obtaining a loss value based on a difference between a video feature of each third sample video and a text feature of a text label corresponding to the third sample video;

[0046] Based on the obtained loss value, the model parameters of the multimodal large model of the initial structure are adjusted until the preset convergence conditions are reached.

[0047] In some embodiments, the training process of the micro-motion recognition model includes the following steps:

[0048] Obtaining a first sample video and a text label corresponding to the first sample video;

[0049] Inputting the text label corresponding to each first sample video into the text feature extraction network in the micro-motion recognition model of the initial structure to obtain the text feature of the text label corresponding to the first sample video;

[0050] For each first sample video, the first sample video is input into a visual feature extraction network in a micro-motion recognition model of an initial structure to obtain a first sample video feature corresponding to the first sample video; wherein the first sample video feature includes a plurality of first sample feature vectors, the plurality of first sample feature vectors correspond one-to-one to video frames included in the first sample video, and the position sequence of the plurality of first sample feature vectors in the first sample video feature is consistent with the time sequence of the corresponding video frames in the first sample video;

[0051] Calculate the adjustment amount corresponding to each first sample feature vector according to the position sequence of the first sample feature vector in the first sample video feature;

[0052] Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain a second sample video feature corresponding to the first sample video;

[0053] For each first sample video, a loss value is obtained based on a difference between a feature of a second sample video corresponding to the first sample video and a text feature of a text label corresponding to the first sample video;

[0054] Based on the obtained loss value, the model parameters of the visual feature extraction network in the micro-motion recognition model of the initial structure are adjusted until the preset convergence conditions are reached.

[0055] In some embodiments, using each adjustment amount to adjust the first sample feature vector corresponding to the first sample video feature to obtain the second sample video feature corresponding to the first sample video includes:

[0056] Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain an adjusted first sample video feature;

[0057] Performing convolution processing on the adjusted first sample video features to obtain multiple feature maps with the same dimensions;

[0058] For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain the second sample feature vector corresponding to the position;

[0059] The obtained second sample feature vectors are input into the sequence network in the micro-motion recognition model, and the feature vectors output by the sequence network at multiple moments are combined to obtain the second sample video features corresponding to the first sample video.

[0060] In some embodiments, the process of obtaining the text label corresponding to the sample video includes the following steps:

[0061] For each sample video, the sample video and the labeled prompt word are input into a pre-trained video understanding model to obtain a result output by the video understanding model; wherein the labeled prompt word indicates that the video understanding model determines the micro-motion of a specified object in the input sample video;

[0062] Based on the results output by the video understanding model, a text label corresponding to the sample video is obtained.

[0063] In some embodiments, the multimodal large model of the initial structure is obtained by pre-training based on the fourth sample video and the text label corresponding to the fourth sample video; the text label corresponding to each fourth sample video is used to describe the fourth sample video;

[0064] The method of adjusting the model parameters of the multimodal large model of the initial structure based on the obtained loss value until a preset convergence condition is reached includes:

[0065] Based on the obtained loss value, the model parameters of the specified network layer of the visual feature extraction network in the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached;

[0066] Among them, among the network layers included in the visual feature extraction network in the multimodal large model of the initial structure, the specified network layer is located after other network layers in order; when adjusting the model parameters of the specified network layer, the model parameters of the text feature extraction network in the multimodal large model of the initial structure are kept unchanged.

[0067] In some embodiments, the determining whether a specified object in the video to be identified has a micro-motion represented by the preset prompt word based on a matching result between the second video feature to be identified and the text feature of the preset prompt word includes:

[0068] Calculating the similarity between the second video feature to be identified and the text feature of the preset prompt word;

[0069] If the calculated similarity is greater than a first threshold, it is determined that the specified object in the video to be identified has a micro-motion represented by the preset prompt word;

[0070] If the calculated similarity is not greater than the first threshold, it is determined that the specified object in the to-be-identified video does not have the micro-motion represented by the preset prompt word.

[0071] According to a second aspect of an embodiment of the present application, a micro-motion recognition device is provided, the device comprising:

[0072] A module for acquiring videos to be identified, used to acquire videos to be identified that contain a specified object;

[0073] A visual feature extraction module, used for inputting each video frame in the video to be identified into a visual feature extraction network in a pre-trained micro-motion recognition model to obtain a first video feature to be identified; wherein the first video feature to be identified includes a plurality of first feature vectors to be identified, the plurality of first feature vectors to be identified correspond one-to-one to the video frames included in the video to be identified, and the position sequence of each first feature vector to be identified in the first video feature to be identified is consistent with the time sequence of the corresponding video frame in the video to be identified;

[0074] An adjustment amount calculation module, used for calculating the adjustment amount corresponding to each first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified;

[0075] An adjustment module, configured to adjust the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount, so as to obtain a second video feature to be identified;

[0076] A matching module is used to determine whether a specified object in the video to be identified has a micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word; wherein the text feature of the preset prompt word is obtained by extracting features of the preset prompt word based on the text feature extraction network in the micro-motion recognition model; the micro-motion recognition model is trained based on a first sample video and the text label corresponding to the first sample video, and each text label corresponding to the first sample video represents a micro-motion existing in the specified object in the first sample video.

[0077] In some embodiments, the adjustment amount calculation module is specifically used to:

[0078] According to the position sequence of each first feature vector to be identified in the first video feature to be identified, the adjustment amount corresponding to the first feature vector to be identified is calculated according to the following formula:

[0079] ;

[0080] The value range of i is between 1 and ; Indicates An adjustment amount corresponding to a first feature vector to be identified; represents the total number of video frames contained in the video to be identified; d represents the dimension of the first feature vector to be identified.

[0081] In some embodiments, the micro-motion recognition model further comprises a sequence network;

[0082] The adjustment module comprises:

[0083] A first adjustment submodule, configured to adjust the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount, to obtain an adjusted first video feature to be identified;

[0084] A first convolution submodule is used to perform convolution processing on the adjusted first video feature to be identified to obtain multiple feature maps with the same dimension;

[0085] A second feature vector acquisition submodule to be identified is used to obtain the value of the element at each position in the obtained feature map, and obtain the second feature vector to be identified corresponding to the position;

[0086] The second to-be-recognized video feature acquisition submodule is used to input each obtained second to-be-recognized feature vector into the sequence network, and combine the feature vectors output by the sequence network at multiple moments to obtain the second to-be-recognized video feature.

[0087] In some embodiments, the sequence network comprises a hidden layer and an output layer;

[0088] The second to-be-identified video feature acquisition submodule is specifically used to:

[0089] Using the weight of the sequence network at the current moment, a weighted sum of the second feature vectors to be identified is calculated as the weighted feature vector at the current moment; wherein the weight of the sequence network at the current moment is obtained based on the second feature vectors to be identified and the feature vector output by the hidden layer at the previous moment;

[0090] Inputting the feature vector output by the output layer at the last moment, the weighted feature vector at the current moment, and the feature vector output by the hidden layer at the last moment into the hidden layer, to obtain the feature vector output by the output layer at the current moment, and the feature vector output by the hidden layer at the current moment;

[0091] Returning to the step of using the weights of the sequence network at the current moment to calculate the weighted sum of the second feature vectors to be identified as the weighted feature vector at the current moment, until a specified number of feature vectors output by the output layer are obtained;

[0092] The specified number of feature vectors output by the output layer are combined to obtain the second video features to be identified.

[0093] In some embodiments, the preset prompt words include preset positive prompt words, and a positive prompt word indicates a positive micro-motion existing in a specified object when a specified event is triggered;

[0094] The device also includes:

[0095] A designated event determination module is used to determine whether the designated object in the video to be identified has a micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word. If the designated object in the video to be identified has the positive example micro-motion, then determine that the designated object in the video to be identified triggers the designated event.

[0096] In some embodiments, the preset prompt word includes a preset negative prompt word, and the negative prompt word represents a negative example micro-motion other than the positive example micro-motion;

[0097] The designated event determination module is specifically used to:

[0098] If the specified object in the video to be identified has the positive micro-motion and does not have the negative micro-motion, it is determined that the specified object in the video to be identified triggers the specified event.

[0099] In some embodiments, the preset prompt word is obtained based on the following steps:

[0100] Obtain each candidate prompt word and a second sample video; wherein the candidate prompt word includes: a plurality of candidate positive prompt words and a plurality of candidate negative prompt words; the second sample video includes a plurality of second positive sample videos and a plurality of second negative sample videos, a specified object in any second positive sample video triggers the specified event, and a specified object in any second negative sample video does not trigger the specified event;

[0101] Input each candidate prompt word into a text feature extraction network in a pre-trained multimodal large model to obtain text features of each candidate prompt word;

[0102] Inputting each second sample video into a visual feature extraction network in a pre-trained multimodal large model to obtain video features of each second sample video;

[0103] For each second sample video, calculating the similarity between the text features of each candidate prompt word and the video features of the second sample video;

[0104] Based on the obtained similarity, a prompt word that matches the second positive sample video and does not match the second negative sample video is determined from each candidate positive prompt word as a preset positive prompt word; and a prompt word that does not match the second positive sample video and matches the second negative sample video is determined from each candidate negative prompt word as a preset negative prompt word.

[0105] In some embodiments, the text feature extraction network in the micro-motion recognition model is a text feature extraction network in a pre-trained multimodal large model.

[0106] In some embodiments, the training process of the multimodal large model includes the following steps:

[0107] Obtaining a third sample video and a text label corresponding to the third sample video; wherein the text label corresponding to each third sample video indicates a micro-motion of a specified object in the third sample video;

[0108] Input each third sample video into the visual feature extraction network in the multimodal large model of the initial structure to obtain the video features of the third sample video;

[0109] Inputting the text label corresponding to each third sample video into the text feature extraction network in the multimodal large model of the initial structure to obtain the text feature of the text label corresponding to the third sample video;

[0110] Obtaining a loss value based on a difference between a video feature of each third sample video and a text feature of a text label corresponding to the third sample video;

[0111] Based on the obtained loss value, the model parameters of the multimodal large model of the initial structure are adjusted until the preset convergence conditions are reached.

[0112] In some embodiments, the training process of the micro-motion recognition model includes the following steps:

[0113] Obtaining a first sample video and a text label corresponding to the first sample video;

[0114] Inputting the text label corresponding to each first sample video into the text feature extraction network in the micro-motion recognition model of the initial structure to obtain the text feature of the text label corresponding to the first sample video;

[0115] For each first sample video, the first sample video is input into a visual feature extraction network in a micro-motion recognition model of an initial structure to obtain a first sample video feature corresponding to the first sample video; wherein the first sample video feature includes a plurality of first sample feature vectors, the plurality of first sample feature vectors correspond one-to-one to video frames included in the first sample video, and the position sequence of the plurality of first sample feature vectors in the first sample video feature is consistent with the time sequence of the corresponding video frames in the first sample video;

[0116] Calculate the adjustment amount corresponding to each first sample feature vector according to the position sequence of the first sample feature vector in the first sample video feature;

[0117] Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain a second sample video feature corresponding to the first sample video;

[0118] For each first sample video, a loss value is obtained based on a difference between a feature of a second sample video corresponding to the first sample video and a text feature of a text label corresponding to the first sample video;

[0119] Based on the obtained loss value, the model parameters of the visual feature extraction network in the micro-motion recognition model of the initial structure are adjusted until the preset convergence conditions are reached.

[0120] In some embodiments, using each adjustment amount to adjust the first sample feature vector corresponding to the first sample video feature to obtain the second sample video feature corresponding to the first sample video includes:

[0121] Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain an adjusted first sample video feature;

[0122] Performing convolution processing on the adjusted first sample video features to obtain multiple feature maps with the same dimensions;

[0123] For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain the second sample feature vector corresponding to the position;

[0124] The obtained second sample feature vectors are input into the sequence network in the micro-motion recognition model, and the feature vectors output by the sequence network at multiple moments are combined to obtain the second sample video features corresponding to the first sample video.

[0125] In some embodiments, the process of obtaining the text label corresponding to the sample video includes the following steps:

[0126] For each sample video, the sample video and the labeled prompt word are input into a pre-trained video understanding model to obtain a result output by the video understanding model; wherein the labeled prompt word indicates that the video understanding model determines the micro-motion of a specified object in the input sample video;

[0127] Based on the results output by the video understanding model, a text label corresponding to the sample video is obtained.

[0128] In some embodiments, the multimodal large model of the initial structure is obtained by pre-training based on the fourth sample video and the text label corresponding to the fourth sample video; the text label corresponding to each fourth sample video is used to describe the fourth sample video;

[0129] The method of adjusting the model parameters of the multimodal large model of the initial structure based on the obtained loss value until a preset convergence condition is reached includes:

[0130] Based on the obtained loss value, the model parameters of the specified network layer of the visual feature extraction network in the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached;

[0131] Among them, among the network layers included in the visual feature extraction network in the multimodal large model of the initial structure, the specified network layer is located after other network layers in order; when adjusting the model parameters of the specified network layer, the model parameters of the text feature extraction network in the multimodal large model of the initial structure are kept unchanged.

[0132] In some embodiments, the matching module is specifically used to:

[0133] Calculating the similarity between the second video feature to be identified and the text feature of the preset prompt word;

[0134] If the calculated similarity is greater than a first threshold, it is determined that the specified object in the video to be identified has a micro-motion represented by the preset prompt word;

[0135] If the calculated similarity is not greater than the first threshold, it is determined that the specified object in the to-be-identified video does not have the micro-motion represented by the preset prompt word.

[0136] According to a third aspect of the embodiments of the present application, an electronic device is provided, including:

[0137] Memory, used to store computer programs;

[0138] The processor is used to implement any of the above-mentioned micro-motion recognition methods when executing the program stored in the memory.

[0139] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the micro-motion recognition methods described above is implemented.

[0140] According to another aspect of the embodiments of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the micro-motion recognition methods described above.

[0141] Beneficial effects of the embodiments of the present application:

[0142] Based on a micro-motion recognition method provided by an embodiment of the present application, after obtaining a video to be recognized, each video frame in the video to be recognized can be input into a visual feature extraction network in a pre-trained micro-motion recognition model to obtain a first video feature to be recognized. And the first video feature to be recognized is composed of each first feature vector to be recognized according to the time sequence of the corresponding video frame in the video to be recognized. Therefore, the first video feature to be recognized can characterize the visual features of each video frame in the video to be recognized, and can also characterize the time sequence between each video frame. Furthermore, the adjustment amount corresponding to the first feature vector to be recognized can be calculated according to the position sequence of each first feature vector to be recognized in the first video feature to be recognized, so as to adjust the first feature vector to be recognized in the first video feature to be recognized, and obtain the second video feature to be recognized. In this way, the second video feature to be recognized further combines the visual features of each video frame in the video to be recognized and the time sequence relationship between the video frames, which can better reflect the picture changes of the video to be recognized, and can also better reflect the micro-motions of the specified object in the video through the picture changes of the video to be recognized. It is understandable that, since the micro-motion recognition model is trained based on the first sample video and the text label corresponding to the first sample video, the text label corresponding to each first sample video represents the micro-motion of the specified object in the first sample video. Therefore, the micro-motion recognition model can learn the relationship between the video content and the micro-motion of the specified object in the video. And the second video feature to be identified is extracted and adjusted based on the visual feature extraction network in the micro-motion recognition model, and the text feature of the preset prompt word is extracted based on the text feature extraction network in the micro-motion recognition model. The corresponding prompt word is obtained by extracting the preset prompt word. Therefore, after obtaining the second video feature to be identified, the second video feature to be identified can be matched with the text feature of the preset prompt word, so as to determine whether the specified object in the video to be identified has the micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word. In this way, it is possible to recognize the micro-motion of the specified object in the video to be identified, and ensure the accuracy of micro-motion recognition.

[0143] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0145] Figure 1A first flow chart of a micro-motion recognition method provided in an embodiment of the present application;

[0146] Figure 2 A second flow chart of the micro-motion recognition method provided in an embodiment of the present application;

[0147] Figure 3 A schematic diagram of the structure of a sequence network provided in an embodiment of the present application;

[0148] Figure 4 A third flow chart of the micro-motion recognition method provided in an embodiment of the present application;

[0149] Figure 5 A flowchart for obtaining a preset prompt word provided in an embodiment of the present application;

[0150] Figure 6 A flowchart of a micro-motion recognition model for training provided in an embodiment of the present application;

[0151] Figure 7 A schematic diagram of performing micro-motion recognition provided in an embodiment of the present application;

[0152] Figure 8 A schematic diagram of the structure of a micro-motion recognition model provided in an embodiment of the present application;

[0153] Fig. 9 A schematic diagram of a process for calculating similarity provided in an embodiment of the present application;

[0154] Fig.10 A structural diagram of a model optimization device provided in an embodiment of the present application;

[0155] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0156] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0157] With the development of computer vision technology, the demand for human motion recognition is becoming more and more widespread, especially the recognition of human micro-motions has become a challenging task. Human micro-motions refer to subtle body movements or facial expressions of the human body. For example, in a driving scenario, it is necessary to identify whether the driver has micro-motions such as making a phone call, yawning, closing eyes, etc.; for another example, in a smart device control scenario, it is necessary to identify micro-motions such as gestures made by the user.

[0158] Micro-motion recognition usually involves the precise capture and analysis of subtle body movements or facial expressions. In this application, advanced image processing and machine learning techniques can be used to monitor and identify tiny movements in real time. The difficulties in micro-motion recognition mainly include: weak signals (micro-motions are small in amplitude, low in signal strength, easily submerged by noise, and difficult to extract effective signals), difficulty in data collection (factors such as lighting changes, occlusions, and equipment accuracy in the actual environment will affect data quality), large individual differences (micro-motion characteristics of different individuals may vary significantly, and it is challenging to establish a general model), and poor model generalization ability (in different scenarios, the recognition effect of the same micro-motion recognition model often varies greatly).

[0159] It can be seen that the recognition of micro-motions is difficult due to the small amplitude, fast changes, and individual differences of micro-motions. Therefore, how to realize micro-motion recognition and ensure the accuracy of micro-motion recognition has become a problem that needs to be solved urgently.

[0160] An embodiment of the present application provides a micro-motion recognition method, which can be applied to electronic devices.

[0161] For example, in the above-mentioned driving scenario, the in-car driving recorder can obtain a video containing the driver. Accordingly, the micro-motion recognition method provided in the embodiment of the present application can identify whether the driver in the acquired video has micro-motions such as closing his eyes, yawning, and lowering his head. Subsequently, it can be determined whether the driver in the video is fatigued driving based on the recognition result. Furthermore, when the driver triggers a fatigue driving event (i.e., triggers a specified event in this application), the driver can be reminded to stop and rest. Accordingly, in this driving scenario, the electronic device can be the above-mentioned in-car driving recorder. Alternatively, the electronic device can also be a cloud server, and the cloud server can communicate with the in-car driving recorder.

[0162] For another example, in the above-mentioned smart device control scenario, the camera of the smart device can shoot a video containing the user's hands. Accordingly, the micro-motion recognition method provided in the embodiment of the present application can identify whether the user's hands in the acquired video have micro-motions such as clenching a fist, waving hands left and right. Subsequently, the smart device can perform corresponding control operations according to the recognition results. For example, when it is recognized that the user's hand has a micro-motion of clenching a fist, the smart device can control the machine to shut down. Accordingly, in this smart device control scenario, the electronic device can be the above-mentioned smart device.

[0163] See also Figure 1 , Figure 1 The first flow chart of the micro-motion recognition method provided in the embodiment of the present application comprises the following steps:

[0164] S101: Obtain a video to be identified that contains a specified object.

[0165] S102: Input each video frame in the video to be identified into a visual feature extraction network in a pre-trained micro-motion recognition model to obtain a first video feature to be identified.

[0166] Among them, the first video feature to be identified includes multiple first feature vectors to be identified, and the multiple first feature vectors to be identified correspond one-to-one to the video frames contained in the video to be identified, and the position sequence of each first feature vector to be identified in the first video feature to be identified is consistent with the time sequence of the corresponding video frame in the video to be identified.

[0167] S103: Calculate the adjustment amount corresponding to each first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified.

[0168] S104: using each adjustment amount, adjusting the first feature vector to be identified corresponding to the first video feature to be identified, to obtain a second video feature to be identified.

[0169] S105: Based on the matching result of the second video feature to be identified and the text feature of the preset prompt word, determine whether the specified object in the video to be identified has a micro-motion represented by the preset prompt word.

[0170] Among them, the text features of the preset prompt words are obtained by extracting features of the preset prompt words based on the text feature extraction network in the micro-motion recognition model; the micro-motion recognition model is trained based on the first sample video and the text labels corresponding to the first sample video, and the text labels corresponding to each first sample video represent the micro-motions of the specified object in the first sample video.

[0171] Based on the above processing, after obtaining the video to be identified, each video frame in the video to be identified can be input into the visual feature extraction network in the pre-trained micro-motion recognition model to obtain the first video feature to be identified. And the first video feature to be identified is composed of each first feature vector to be identified according to the time sequence of the corresponding video frame in the video to be identified. Therefore, the first video feature to be identified can characterize the visual features of each video frame in the video to be identified, and can also characterize the time sequence between each video frame. Furthermore, according to the position sequence of each first feature vector to be identified in the first video feature to be identified, the adjustment amount corresponding to the first feature vector to be identified can be calculated to adjust the first feature vector to be identified in the first video feature to be identified, and obtain the second video feature to be identified. In this way, the second video feature to be identified further combines the visual features of each video frame in the video to be identified and the time sequence relationship between the video frames, which can better reflect the picture changes of the video to be identified, and can also better reflect the micro-motions of the specified object in the video through the picture changes of the video to be identified. It is understandable that, since the micro-motion recognition model is trained based on the first sample video and the text label corresponding to the first sample video, the text label corresponding to each first sample video represents the micro-motion of the specified object in the first sample video. Therefore, the micro-motion recognition model can learn the relationship between the video content and the micro-motion of the specified object in the video. And the second video feature to be identified is extracted and adjusted based on the visual feature extraction network in the micro-motion recognition model, and the text feature of the preset prompt word is extracted based on the text feature extraction network in the micro-motion recognition model. The corresponding prompt word is obtained by extracting the preset prompt word. Therefore, after obtaining the second video feature to be identified, the second video feature to be identified can be matched with the text feature of the preset prompt word, so as to determine whether the specified object in the video to be identified has the micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word. In this way, it is possible to recognize the micro-motion of the specified object in the video to be identified, and ensure the accuracy of micro-motion recognition.

[0172] In step S101, the electronic device may obtain a video that needs to be identified with micro-motions (ie, the video to be identified), and identify whether a designated object in the video to be identified has micro-motions. The designated object is preset according to actual identification requirements.

[0173] For example, in the above driving scene, the electronic device needs to recognize the micro-movements of the driver's face in the video. Accordingly, in the driving scene, the designated object is the human face.

[0174] For another example, in the above-mentioned smart device control scenario, the electronic device needs to recognize the micro-movements of the user's hands in the video. Accordingly, in the smart device control scenario, the designated object is the human hand.

[0175] With respect to step S102, after obtaining the video to be identified, that is, after obtaining each video frame in the video to be identified, the electronic device can input each video frame in the video to be identified into a visual feature extraction network in a pre-trained micro-motion recognition model to obtain the video feature of the video to be identified (that is, the first video feature to be identified). The training process of the micro-motion recognition model will be described in subsequent embodiments.

[0176] The first video feature to be identified includes multiple feature vectors (i.e., first feature vectors to be identified), and the multiple first feature vectors to be identified correspond one-to-one to the video frames included in the video to be identified, and the position sequence of each first feature vector to be identified in the first video feature to be identified is consistent with the timing of the corresponding video frame in the video to be identified.

[0177] That is to say, a column of the first video features to be identified is the feature vector of a video frame in the video to be identified (ie, the first feature vector to be identified), and the first video features to be identified are composed of the first feature vectors to be identified in the time sequence of the corresponding video frames in the video to be identified.

[0178] Correspondingly, the first video feature to be identified (denoted as ) can be expressed as: .

[0179] in, represents the feature vector of the i-th video frame (i.e., the first feature vector to be identified); n represents the total number of video frames contained in the video to be identified.

[0180] With respect to step S103 and step S104, for each first feature vector to be identified in the first video feature to be identified, the electronic device may calculate the adjustment amount corresponding to the first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified.

[0181] In some embodiments, the adjustment amount corresponding to each first feature vector to be identified can be determined by a periodic function (ie, a sin function and a cos function). Step S103 includes:

[0182] According to the position sequence of each first feature vector to be identified in the first video feature to be identified, the adjustment amount corresponding to the first feature vector to be identified is calculated according to the following formula:

[0183] ;

[0184] in, represents the adjustment amount corresponding to the i-th first feature vector to be identified; the value range of i is between 1 and ; represents the total number of video frames contained in the video to be identified; d represents the dimension of the first feature vector to be identified.

[0185] In the embodiment of the present application, for each first feature vector to be identified, the position order of the first feature vector to be identified in the first video feature to be identified, that is, the column number corresponding to the first feature vector to be identified in the first video feature to be identified. d represents the dimension of the first feature vector to be identified, that is, the number of elements contained in the feature vector of any video frame.

[0186] Based on the above processing, the adjustment amount corresponding to each first feature vector to be identified can be determined by a periodic function (i.e., a sin function and a cos function). In the process of calculating the adjustment amount corresponding to each first feature vector to be identified according to the above periodic function, it can be ensured that the adjustment amount corresponding to each first feature vector to be identified is unique, and it can also be ensured that the adjustment amount corresponding to each first feature vector to be identified can reflect the difference between the positions of each first feature vector to be identified in the first video feature to be identified.

[0187] In addition, due to the nature of the periodic function (i.e., the sin function and the cos function), the calculated adjustment amount corresponding to each first feature vector to be identified is bounded and is between [-1,1]. In this way, the generalization of the calculated adjustment amount can be guaranteed, that is, for videos to be identified containing any number of video frames, the adjustment amount corresponding to each first feature vector to be identified can be calculated by the above formula, which also ensures that the micro-motion recognition method provided by the present application is applicable to videos containing different numbers of video frames.

[0188] In some embodiments, the adjustment amount corresponding to each first feature vector to be identified can also be determined by marking the position with an integer value. That is, the position sequence of each first feature vector to be identified in the first video feature to be identified is used as the adjustment amount corresponding to the first feature vector to be identified. In other words, for the feature vector of the first video frame in the video to be identified (i.e., the first feature vector to be identified in the first column of the first video feature to be identified), the adjustment amount corresponding to the feature vector is 1; for the feature vector of the second video frame in the video to be identified (i.e., the first feature vector to be identified in the second column of the first video feature to be identified), the adjustment amount corresponding to the feature vector is 2, and so on, to obtain the adjustment amount corresponding to each first feature vector to be identified.

[0189] Then, the electronic device can use each adjustment amount to adjust the first feature vector corresponding to the first video feature to be identified to obtain the second video feature to be identified. The following are two ways to obtain the second video feature to be identified:

[0190] In one implementation, the electronic device may adjust the first feature vector to be identified in the first video feature to be identified by using the adjustment amount corresponding to each first feature vector to be identified, and use the adjusted first video feature to be identified as the second video feature to be identified.

[0191] For example, the obtained adjusted first video feature to be identified can be expressed as:

[0192] ;

[0193] in, represents the adjusted first video feature to be identified; the value range of i is between 1 and ; Indicates the total number of video frames contained in the video to be identified.

[0194] In another implementation, the micro-motion recognition model also includes a sequence network. Figure 2 , Figure 2 A second flow chart of the micro-motion recognition method provided in the embodiment of the present application, Figure 1 On the basis of, step S104 comprises:

[0195] S1041: Using each adjustment amount, adjust the first feature vector corresponding to the first video feature to be identified to obtain the adjusted first video feature to be identified.

[0196] S1042: Perform convolution processing on the adjusted first video feature to be identified to obtain multiple feature maps with the same dimension.

[0197] S1043: For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain a second feature vector to be identified corresponding to the position.

[0198] S1044: Input each obtained second feature vector to be identified into the sequence network, and combine the feature vectors output by the sequence network at multiple time points to obtain the second video feature to be identified.

[0199] In the embodiment of the present application, the process of adjusting the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount can refer to the above embodiment, and will not be described in detail here. Accordingly, after obtaining the adjusted first video feature to be identified, the electronic device can perform convolution processing on the adjusted first video feature to be identified to obtain multiple feature maps with the same dimensions. The dimension of the feature map represents the height and width of the feature map.

[0200] For example, the electronic device may input the adjusted first video feature to be recognized into the Encoder part of the Transform model to obtain multiple feature maps with the same dimensions output by the Encoder in the Transform model, wherein the number of feature maps is determined by the number of channels of the output layer of the Encoder part.

[0201] The multiple feature maps with the same dimensions can be expressed as: , where C represents the number of feature maps, Represents the height of a feature map, Indicates the width of a feature map.

[0202] Correspondingly, there is Locations, For each position in the obtained feature map, the electronic device can obtain the value of the element at that position on each feature map to obtain the second feature vector to be identified corresponding to that position. For example, the second feature vector to be identified corresponding to the jth position can be recorded as , =1,2,…, ,Right now, Represents a C-dimensional vector.

[0203] Furthermore, the electronic device can input each obtained second feature vector to be identified into the sequence network, and combine the feature vectors output by the sequence network at multiple times to obtain the second video feature to be identified. The sequence network is a recurrent neural network (RNN). For example, the recurrent neural network can be an LSTM (Long Short-Term Memory).

[0204] Based on the above processing, after obtaining the adjusted first video feature to be identified, the electronic device can perform convolution processing on the adjusted first video feature to be identified to extract the deep features of the video to be identified. Furthermore, the feature graphs with multiple dimensions obtained by convolution can be transformed to obtain the second feature vector to be identified corresponding to each position in the feature graph, and each second feature vector to be identified is input into the sequence network, and the feature vectors output by the sequence network at multiple times are combined to obtain the second video feature to be identified. It can be seen from the characteristics of the sequence network that the sequence network is suitable for sequence data of various lengths, which can ensure the stability of processing the second feature vectors to be identified corresponding to different videos, and thus, can improve the scope of application of the micro-motion recognition method provided in the present application.

[0205] In one implementation, the sequence network includes a hidden layer and an output layer. Step S1044 includes:

[0206] Step 1: Using the weight of the sequence network at the current moment, calculate the weighted sum of the second feature vectors to be identified as the weighted feature vector at the current moment.

[0207] The weight of the sequence network at the current moment is obtained based on each second feature vector to be identified and the feature vector output by the hidden layer at the previous moment.

[0208] Step 2: Input the feature vector output by the output layer at the previous moment, the weighted feature vector at the current moment, and the feature vector output by the hidden layer at the previous moment into the hidden layer to obtain the feature vector output by the output layer at the current moment and the feature vector output by the hidden layer at the current moment; return to execute step 1 until the specified number of feature vectors output by the output layer is obtained.

[0209] Step 3: Combine the specified number of feature vectors output by the output layer to obtain the second video feature to be identified.

[0210] In the embodiment of the present application, the weight of the sequence network represents: the weight of the Attention (attention mechanism) part in the sequence network.

[0211] In the weight of the sequence network at the current time (denoted as t), the weight of the second feature vector to be identified corresponding to the jth position can be expressed as: .

[0212] in, , represents the hidden layer output at the previous moment, represents the second feature vector to be identified corresponding to the jth position. That is to say, the weight of the second feature vector to be identified corresponding to the jth position at the current moment is learned through the attention mechanism based on the second feature vector to be identified corresponding to the jth position and the feature vector output by the hidden layer at the previous moment.

[0213] Correspondingly, the weighted sum of the second feature vectors to be identified is calculated using the weight of the current sequence network (denoted as ) can be expressed as:

[0214] ;

[0215] Furthermore, the electronic device can input the feature vector output by the output layer at the previous moment, the weighted feature vector at the current moment, and the feature vector output by the hidden layer at the previous moment into the hidden layer to obtain the feature vector output by the output layer at the current moment, and the feature vector output by the hidden layer at the current moment.

[0216] For example, the feature vector output by the output layer at the current moment can be expressed as:

[0217] ;

[0218] in, Represents the feature vector of the output layer output at the current moment; Indicates the current moment, Indicates the previous moment; Represents the feature vector output by the output layer at the previous moment; The feature vector representing the output of the hidden layer at the previous moment; Represents the weighted feature vector at the current moment, and RNN represents the sequence network.

[0219] After obtaining the feature vector output by the output layer at the current moment and the feature vector output by the hidden layer at the current moment, the electronic device can return to execute the above steps 1 to 2 until a specified number of feature vectors output by the output layer are obtained. It can be understood that during the first execution of the above step 2, , It can be an initialized value. For example, the initialized value can be a random value.

[0220] Accordingly, the electronic device can combine the specified number of feature vectors output by the output layer to obtain the second video feature to be identified. For example, the specified number of feature vectors output by the output layer can be combined into a matrix according to the order of the time when each feature vector is output, that is, the second video feature to be identified is obtained. Among them, one column of the second video feature to be identified corresponds to one feature vector output by the output layer.

[0221] See also Figure 3 , Figure 3 A schematic diagram of the structure of a sequence network provided in an embodiment of the present application. Figure 3 In the equation, t represents the current time, t-1 represents the previous time before the current time, and t+1 represents the next time after the current time. Represents the feature vector of the hidden layer output at the previous moment, Represents the feature vector of the hidden layer output at the current moment, The feature vector representing the output of the hidden layer at the next moment. Represents the feature vector output by the output layer at the previous moment, Represents the feature vector output by the output layer at the current moment, Represents the feature vector output by the output layer at the next moment. represents the weight of the second feature vector to be identified corresponding to the first position at the current moment, represents the weight of the second feature vector to be identified corresponding to the second position at the current moment, Indicates the weight of the second feature vector to be identified corresponding to the Tth position at the current moment. represents the weight of the second feature vector to be identified corresponding to the first position at the next moment of the current moment, represents the weight of the second feature vector to be identified corresponding to the second position at the next moment of the current moment, Indicates the weight of the second feature vector to be identified corresponding to the T-th position at the next moment of the current moment. Represents: the weighted sum of the second feature vectors to be identified is calculated using the weight of the sequence network at the current moment. It means: using the weight of the next moment sequence network at the current moment, the weighted sum of the second feature vectors to be identified is calculated. represents the second feature vector to be identified corresponding to the first position; represents the second feature vector to be identified corresponding to the second position; represents the second feature vector to be identified corresponding to the Tth position. , Represents the height of a feature map, Indicates the width of a feature map.

[0222] and and The connection relationship between and The connection relationship between them means that for any moment, the input of the sequence network at that moment includes: the feature vector output by the hidden layer at the previous moment, the feature vector output by the output layer at the previous moment, and the weighted sum of the second feature vectors to be identified obtained by calculating the weights of the sequence network at that moment.

[0223] and and The connection relationship between and The connection relationship between them means that for any moment, the weight of the second feature vector to be identified corresponding to the j-th position at that moment is the feature vector to be identified corresponding to the j-th position and the feature vector output by the hidden layer at the previous moment, which is learned through the attention mechanism.

[0224] In this way, the electronic device can process the obtained second feature vectors to be identified through the sequence network in the micro-motion recognition model, and because the loop structure of the sequence network can effectively learn the relationship between time and feature changes in the sequence data (i.e., each second feature vector to be identified), the second video features to be identified obtained through the sequence network processing can accurately reflect the changes between each video frame in the video, which is conducive to the recognition of micro-motions of the specified object.

[0225] For step S105, after obtaining the second video feature to be identified corresponding to the video to be identified, the electronic device can match the second video feature to be identified with the text feature of each preset prompt word. Furthermore, the electronic device can determine the micro-motion of the specified object in the video to be identified based on the matching result of the second video feature to be identified with the text feature of each preset prompt word.

[0226] The preset prompt word may be a text describing a micro-motion of the specified object. The process of obtaining the text feature of the preset prompt word will be described in the subsequent embodiments.

[0227] In one implementation, step S105 includes:

[0228] S1051: Calculate the similarity between the second video feature to be identified and the text feature of the preset prompt word.

[0229] S1052: If the calculated similarity is greater than the first threshold, it is determined that the specified object in the video to be identified has a micro-motion represented by a preset prompt word.

[0230] S1053: If the calculated similarity is not greater than the first threshold, it is determined that the specified object in the video to be identified does not have a micro-motion represented by the preset prompt word.

[0231] In an embodiment of the present application, the preset prompt word may be one or more text features for each preset prompt word. After acquiring the second video feature to be identified corresponding to the video to be identified, the electronic device may calculate the similarity between the second video feature to be identified and the text feature of the preset prompt word.

[0232] For any preset prompt word, the higher the calculated similarity, the greater the probability that the specified object in the video to be identified has the micro-motion represented by the preset prompt word. Correspondingly, the lower the calculated similarity, the lower the probability that the specified object in the video to be identified has the micro-motion represented by the preset prompt word.

[0233] Therefore, if the calculated similarity is greater than the first threshold, the electronic device can determine that the specified object in the video to be identified has a micro-motion represented by the preset prompt word. If the calculated similarity is not greater than the first threshold, the electronic device can determine that the specified object in the video to be identified does not have a micro-motion represented by the preset prompt word.

[0234] The similarity between the second video feature to be identified and the text feature of each preset prompt word may be obtained through normalization, that is, the value range of the similarity is between [0, 1]. Accordingly, the first threshold may be set by a technician based on actual experience, for example, the first threshold may be 0.7.

[0235] For example, in the above driving scene, the preset prompt word is "the driver closes his eyes". If the similarity between the second video feature to be identified and the text feature of the preset prompt word is greater than the first threshold, the electronic device can determine that the driver in the video to be identified has a micro-action of closing his eyes.

[0236] Based on the above processing, the electronic device can calculate the similarity between the features of different modalities, that is, calculate the similarity between the second video feature to be identified and the text feature of each preset prompt word. Accordingly, the similarity can represent the probability of the specified object in the video to be identified having the micro-motion represented by the preset prompt word in a quantitative form. In this way, it can accurately determine whether the specified object in the video to be identified has the micro-motion represented by the preset prompt word, thereby improving the efficiency of micro-motion recognition.

[0237] In addition, the micro-motion recognition model in this application has the ability to understand the semantics of the text and match it with the video content. Accordingly, it is possible to establish an association between the visual features of the video and the text description, and realize cross-modal retrieval from text to video. In other words, the multimodal large model can establish associations between different modalities. Through joint learning, the model can use auxiliary information of other modalities when processing a certain modal information, improve the accuracy and depth of micro-motion recognition, and strengthen the processing ability of video signals. And the micro-motion recognition model in this application supports real-time processing of large amounts of video data and has the ability to process large-scale data.

[0238] In the present application, the electronic device can further determine whether the designated object triggers a designated event based on the micro-movement represented by the preset prompt word of the designated object in the video to be identified.

[0239] In some embodiments, the preset prompt words include preset positive prompt words, and a positive prompt word indicates a positive micro-motion of a specified object when a specified event is triggered. Figure 4 , Figure 4 A third flow chart of the micro-motion recognition method provided in the embodiment of the present application, Figure 1 On the basis of, after step S105, the method further includes:

[0240] S106: If the specified object in the video to be identified has a positive micro-motion, it is determined that the specified object in the video to be identified triggers a specified event.

[0241] It is understandable that in different scenarios where micro-motion recognition is required, the specified event required to determine whether the specified object is triggered is also different. Accordingly, the micro-motions (i.e., positive micro-motions) that indicate the existence of the specified object when the specified event is triggered may also be different.

[0242] In the embodiment of the present application, the above-mentioned driving scene is used as an example to illustrate that the driver may endanger the safety of himself and others when driving fatigued. Therefore, the electronic device needs to determine whether the driver (i.e., the specified object) in the video to be identified triggers a fatigue driving event (i.e., the specified event). It can be seen from practical experience that when the driver is driving fatigued, there are usually micro-movements such as closing eyes, yawning, and lowering the head. Therefore, in this scene, the positive example micro-movements can be micro-movements such as closing eyes, yawning, and lowering the head. Correspondingly, the preset positive prompt words can include: "driver closes eyes", "driver yawns", "driver lowers head", etc.

[0243] Alternatively, taking the above-mentioned driving scenario as an example, a driver distracted by playing with a mobile phone while driving may endanger the safety of himself and others. Therefore, the electronic device needs to determine whether the driver (i.e., the specified object) in the video to be identified triggers a mobile phone playing event (i.e., the specified event). From practical experience, it can be known that when a driver plays with a mobile phone, there are usually micro-movements such as lowering the head, looking at the mobile phone, and making a phone call. Therefore, in this scenario, the positive micro-movements can be micro-movements such as lowering the head, looking at the mobile phone, and making a phone call. Correspondingly, the preset positive prompt words can include: "The driver lowers his head", "The driver looks at the mobile phone", "The driver makes a phone call", etc.

[0244] The specific process of selecting the preset positive prompt word will be described in the subsequent embodiments.

[0245] Correspondingly, when it is determined according to the above steps S101 to S105 that the designated object in the video to be identified has a positive micro-motion, the electronic device can determine that the designated object in the video to be identified triggers a designated event.

[0246] Continuing with the above example, if the electronic device determines that the driver in the video to be identified has positive micro-movements such as closing eyes, yawning, and lowering head according to the above steps S101 to S105, it can be determined that the driver in the video to be identified has triggered a fatigue driving event.

[0247] In one implementation, the electronic device may determine that the specified object in the video to be identified triggers the specified event when the ratio of the positive micro-motions of the specified object in the video to be identified is greater than a second threshold.

[0248] For example, the number of preset positive prompt words is 3, that is, the positive micro-motion includes three micro-motions. If the second threshold is 0.5, it means that when the number of positive micro-motions of the specified object in the video to be identified is not less than 2, it is determined that the specified object in the video to be identified triggers the specified event. In this way, the recognition results corresponding to multiple preset prompt words can be further combined to improve the accuracy of the specified event detection.

[0249] Based on the above processing, in different scenarios where micro-motion recognition is required, the micro-motions to be recognized can be set according to actual needs, and then, based on the micro-motion recognition results, it is further determined whether the specified object in the video to be recognized triggers the specified event. In this way, the results of micro-motion recognition can be fully utilized and combined with specific usage scenarios to improve the practicality of micro-motion recognition.

[0250] In some embodiments, the preset prompt words may further include preset negative prompt words on the basis that the preset prompt words include preset positive prompt words, and the negative prompt words represent negative micro-movements other than positive micro-movements. Accordingly, step S106 includes:

[0251] If the specified object in the video to be identified has positive micro-motions and does not have negative micro-motions, it is determined that the specified object in the video to be identified triggers the specified event.

[0252] It is understandable that in different scenarios where micro-motion recognition is required, the specified event required to determine whether the specified object is triggered is also different. Accordingly, the negative micro-motions other than the positive micro-motions that exist when the specified object triggers the specified event may also be different.

[0253] In the embodiment of the present application, the above driving scene is used as an example for explanation. It is known from practical experience that the micro-movements of the driver such as looking down and squinting cannot usually be used as the basis for judging whether the driver is driving fatigued. In addition, the micro-movements of the driver such as singing and talking usually indicate that the driver has not triggered a fatigue driving event. Therefore, in this scene, the negative micro-movements can be micro-movements such as looking down, squinting, and talking. Accordingly, the preset negative prompt words can include: "the driver looks down", "the driver squints", "the driver talks", etc.

[0254] The specific process of selecting the preset negative prompt word will be described in the subsequent embodiments.

[0255] That is, if the designated object in the video to be identified has the above-mentioned negative micro-motion, it indicates that the designated object in the video to be identified has not triggered the designated event, or there is a micro-motion that interferes with the determination of whether the designated object in the video to be identified has triggered the designated event. Therefore, if the electronic device determines that the designated object in the video to be identified has the negative micro-motion according to the above steps S101 to S105, it cannot determine whether the designated object in the video to be identified has triggered the designated event.

[0256] Correspondingly, if the specified object in the video to be identified has a positive micro-motion and does not have a negative micro-motion, it is determined that the specified object in the video to be identified triggers the specified event.

[0257] Based on the above processing, the electronic device can combine the recognition results for the positive micro-motions and the recognition results for the negative micro-motions to jointly determine whether the designated object in the video to be recognized triggers the designated event. In this way, it is possible to avoid misjudgment of the detection result of the designated event when the designated object in the video to be recognized has negative micro-motions that indicate that the designated event has not been triggered, and negative micro-motions that may interfere with the determination of whether the designated object in the video to be recognized triggers the designated event, thereby further improving the accuracy of the detection of the designated event.

[0258] In addition, the electronic device can improve the level of model training during the training of the micro-motion recognition model by setting preset positive prompt words and preset negative prompt words, thereby improving the accuracy of micro-motion recognition.

[0259] In some embodiments, see Figure 5 , Figure 5 A flowchart of obtaining a preset prompt word is provided in an embodiment of the present application. The preset prompt word is obtained based on the following steps:

[0260] S501: Obtain each candidate prompt word and a second sample video.

[0261] Among them, the alternative prompt words include: multiple alternative positive prompt words and multiple alternative negative prompt words; the second sample video includes multiple second positive sample videos and multiple second negative sample videos, the specified object in any second positive sample video triggers the specified event, and the specified object in any second negative sample video does not trigger the specified event.

[0262] S502: Input each candidate prompt word into a text feature extraction network in a pre-trained multimodal large model to obtain text features of each candidate prompt word.

[0263] S503: Input each second sample video into a visual feature extraction network in a pre-trained multimodal large model to obtain video features of each second sample video.

[0264] S504: For each second sample video, calculate the similarity between the text features of each candidate prompt word and the video features of the second sample video.

[0265] S505: Based on the obtained similarity, determine from each candidate positive prompt word a prompt word that matches the second positive sample video and does not match the second negative sample video as a preset positive prompt word; and determine from each candidate negative prompt word a prompt word that does not match the second positive sample video and matches the second negative sample video as a preset negative prompt word.

[0266] In the embodiment of the present application, the candidate prompt words include: a plurality of candidate positive prompt words and a plurality of candidate negative prompt words.

[0267] As mentioned above, in different scenarios where micro-motion recognition is required, the specified events required to determine whether the specified object triggers are also different. Accordingly, the positive prompt words representing positive micro-motions in different scenarios may also be different. Similarly, the negative prompt words representing positive and negative micro-motions in different scenarios may also be different.

[0268] For the current scenario where micro-motion recognition is required, the technician can provide a large number of alternative positive prompt words and alternative negative prompt words in advance, and then, based on the above steps 1 to 5, select a part of the alternative positive prompt words and the alternative negative prompt words as the preset positive prompt words and the preset negative prompt words.

[0269] Taking the above driving scene as an example, the candidate positive prompt words may include: "driver closes eyes", "driver yawns", "driver lowers head", "driver nods", "driver opens mouth", etc. Correspondingly, the candidate negative prompt words may include: "driver looks down", "driver squints", "driver makes phone call", "driver sings", "driver's glasses reflect light", "driver rubs eyes", etc.

[0270] The second sample videos include multiple second positive sample videos and multiple second negative sample videos. A specified object in any second positive sample video triggers a specified event, and a specified object in any second negative sample video does not trigger the specified event.

[0271] Taking the above driving scene as an example, the driver in any second positive sample video has triggered a fatigue driving event, and the driver in any second negative sample video has not triggered a fatigue driving event.

[0272] Accordingly, after obtaining each candidate prompt word and the second sample video, the electronic device can input each candidate prompt word into the text feature extraction network in the pre-trained multimodal large model to obtain the text feature of each candidate prompt word. And, input each second sample video into the visual feature extraction network in the pre-trained multimodal large model to obtain the video feature of each second sample video. The training process of the multimodal large model will be described in the subsequent embodiments.

[0273] It is understandable that, for each second sample video, the electronic device can calculate the similarity between the video features of the second sample video and the text features of each candidate prompt word. If the similarity between a candidate prompt word and a second sample video is greater than the third threshold, it means that the candidate prompt word matches the second sample video.

[0274] For each candidate positive prompt word, if the candidate positive prompt word matches the second positive sample video and does not match the second negative sample video, it can be determined that the candidate positive prompt word is a preset positive prompt word.

[0275] Accordingly, for each candidate positive prompt word, the number of second positive sample videos matching the candidate positive prompt word can be determined as the first matching number of the candidate positive prompt word. Also, the number of second negative sample videos matching the candidate positive prompt word can be determined as the second matching number of the candidate positive prompt word.

[0276] Furthermore, a first specified number of preset positive prompt words can be selected from the candidate positive prompt words according to the first matching number and the second matching number. For example, the electronic device can determine the candidate positive prompt words with a larger first matching number and a smaller second matching number as preset positive prompt words.

[0277] That is, the first matching number of the preset positive prompt word is greater than the other positive prompt words in the candidate positive prompt words except the preset positive prompt word, and the second matching number of the preset positive prompt word is less than the other positive prompt words in the candidate positive prompt words except the preset positive prompt word.

[0278] For each candidate negative prompt word, if the candidate negative prompt word matches the second negative sample video and does not match the second positive sample video, it can be determined that the candidate negative prompt word is a preset negative prompt word.

[0279] Accordingly, for each candidate negative prompt word, the number of second negative sample videos matching the candidate negative prompt word can be determined as the third matching number of the candidate negative prompt word. And the number of second positive sample videos matching the candidate negative prompt word can be determined as the fourth matching number of the candidate negative prompt word.

[0280] Furthermore, a second specified number of preset negative prompt words can be selected from the candidate negative prompt words according to the third matching number and the fourth matching number. For example, the electronic device can determine the candidate negative prompt words with a larger third matching number and a smaller fourth matching number as preset negative prompt words.

[0281] Among them, the third matching number of the preset negative prompt word is greater than other negative prompt words in the candidate negative prompt words except the preset negative prompt word, and the fourth matching number of the preset negative prompt word is less than other negative prompt words in the candidate negative prompt words except the preset negative prompt word.

[0282] Based on the above processing, for the candidate positive prompt words, the candidate positive prompt words with less missed reports in the second positive sample video can be selected as the preset positive prompt words. In addition, based on the matching of the candidate negative prompt words with the second positive sample video, the candidate negative prompt words that are prone to false alarms are determined, and the candidate negative prompt words with the most false alarms are selected. In this way, when the micro-motion recognition of the designated object in the video to be identified is performed in the future according to the selected preset positive prompt words and preset negative prompt words, it can be determined whether the designated object has positive micro-motions and / or negative micro-motions, and it can be avoided that the designated object in the video to be identified has negative micro-motions indicating that the designated event has not been triggered, and the negative micro-motions that may interfere with the judgment of whether the designated object in the video to be identified triggers the designated event. The detection result of the designated event is misjudged, and the accuracy of the detection of the designated event can be further improved.

[0283] Accordingly, for each preset prompt word (including preset positive prompt words and preset negative prompt words), feature extraction can be performed on the preset prompt word based on the text feature extraction network in the micro-motion recognition model to obtain the text feature of the preset prompt word.

[0284] In one implementation, after the micro-motion recognition model is trained, the electronic device can deploy the pre-trained complete micro-motion recognition model. Accordingly, the electronic device can extract features for each preset prompt word based on the text feature extraction network in the micro-motion recognition model to obtain the text features of the preset prompt word.

[0285] In another implementation, the electronic device may deploy only a portion of the pre-trained micro-motion recognition model, that is, the electronic device may deploy only the visual feature extraction network in the pre-trained micro-motion recognition model.

[0286] It is understandable that for a scene, the micro-movements that need to be identified in the scene are usually relatively fixed, so the preset prompt words for the micro-movements indicating the existence of a specified object are also fixed. Accordingly, the electronic device can pre-store the text features of each preset prompt word. Among them, the text features of each preset prompt word can be obtained by pre-extracting features of each preset prompt word based on the text feature extraction network in the micro-movement recognition model, that is, the electronic device can only store the text features of each preset prompt word without deploying the text feature extraction network in the pre-trained micro-movement recognition model.

[0287] In this way, when deploying the micro-motion recognition model on an electronic device (i.e., a platform for deploying the micro-motion recognition model), only a part of the micro-motion recognition model can be deployed, so that the storage space occupied by the micro-motion recognition model in the electronic device can be reduced. In addition, the electronic device can match the text features with the video features based on the pre-recorded text features of each preset prompt word, which can improve the recognition efficiency on the basis of ensuring the recognition of the micro-motions of the specified object in the video to be recognized.

[0288] In some embodiments, the training process of the multimodal large model includes the following steps:

[0289] Step 1: Obtain a third sample video and a text label corresponding to the third sample video.

[0290] The text label corresponding to each third sample video indicates a micro-motion of a specified object in the third sample video.

[0291] Step 2: Input each third sample video into the visual feature extraction network in the multimodal large model of the initial structure to obtain the video features of the third sample video.

[0292] Step 3: Input the text label corresponding to each third sample video into the text feature extraction network in the multimodal large model of the initial structure to obtain the text feature of the text label corresponding to the third sample video.

[0293] Step 4: Obtain a loss value based on the difference between the video feature of each third sample video and the text feature of the text label corresponding to the third sample video.

[0294] Step 5: Based on the obtained loss value, adjust the model parameters of the multimodal large model of the initial structure until the preset convergence conditions are reached.

[0295] In the embodiment of the present application, with respect to the above-mentioned step one, the scene involved in any third sample video is consistent with the scene currently requiring micro-motion recognition.

[0296] Taking the above driving scenario as an example, the third sample video may be: a video collected when the driver triggers a fatigue driving event during historical driving, and / or a video collected when the driver does not trigger a fatigue driving event during historical driving.

[0297] Accordingly, for each third sample video, the electronic device can obtain a text label corresponding to each third sample video, wherein the text label corresponding to each third sample video indicates: a micro-motion of a specified object in the third sample video.

[0298] In one implementation, the process of obtaining the text label corresponding to the sample video (i.e., the third sample video) includes the following steps:

[0299] Step a: For each sample video, the sample video and the labeled prompt words are input into a pre-trained video understanding model to obtain the output result of the video understanding model.

[0300] Among them, the labeled prompt words instruct the video understanding model to determine the micro-movements of the specified object in the input sample video.

[0301] Step b: Based on the output of the video understanding model, obtain the text label corresponding to the sample video.

[0302] In the embodiment of the present application, for each third sample video, the electronic device can input the third sample video and the marking prompt word into a pre-trained video understanding model. For example, the video understanding model can be ChatGPT (Chat Generative Pre-trained Transformer), VideoChat model, etc.

[0303] Correspondingly, the video understanding model can perform the corresponding text generation function according to the input prompt word, and output the text generation result in combination with the content of the input sample video. Among them, the prompt word refers to the initial text or instruction input into the model. This "prompt" will indicate what kind of task the user wants the large language model to perform or what kind of text to generate.

[0304] In this application, the labeling prompt words are used to indicate the micro-movements that the video understanding large model determines the existence of a specified object in the input sample video.

[0305] For example, the labeling prompts may include: "prompt 1: Describe the video in one sentence. (Describe the video in one sentence.) prompt 2: Describe the human actions in the video in one sentence. (Describe the people in the video in one sentence.) prompt 3: Is he tired? (Is he tired?) prompt 4: Is he sleeping? (Is he sleeping?) prompt 5: ...".

[0306] Correspondingly, for each third sample video, the video understanding model can output the answer text corresponding to each labeled prompt word.

[0307] For example, for sample video 1 (the driver in the sample video is on the phone), the process of the video understanding large model outputting the answer text corresponding to each labeled prompt word can be expressed as: "Input 1: Describe the video in one sentence. Output 1: The video is about a man talking on the phone while riding in a car. (This video is about a person talking on the phone while driving.) Input 2: Describe the human actions in the video in one sentence. Output 2: The man in the video talks on the phone while riding in a car. (The man in the video is talking on the phone while riding in a car.) Input 3: Is he tired? Output 3: Yes, the man in the video looks tired. (Yes, the person in the video looks tired.) Input 4: Is he sleeping? Output 4: No, he is not sleeping. (No, he is not sleeping.)"

[0308] For another example, for sample video 2 (the driver in the sample video is yawning), the process of the video understanding large model outputting the answer text corresponding to each labeled prompt word can be expressed as: "Input 1: Describe the video in one sentence. Output 1: The video shows a man getting scared while driving a car while being recorded. (The video shows that a man is scared while driving and is recorded.) Input 2: Describe the human actions in the video in one sentence. Output 2: The man in the video is shown feeling scared while driving a car and being recorded. (The video shows that the man is scared while driving and is recorded.) Input 3: Is he tired? Output 3: The man in the video seems to be tired. (The man in the video looks tired.) Input 4: Is he sleeping? Output 4: Yes, the man in the video is sleeping. (Yes, the man in the video is sleeping.)"

[0309] Furthermore, for each third sample video, the electronic device can obtain a text label corresponding to the sample video based on the result output by the video understanding large model. A sample video and a text label corresponding to the sample video can be called a video-text pair.

[0310] For example, for each third sample video, the electronic device can deduplicate and trim the text content for the third sample video output by the video understanding model, and determine the micro-movements of the specified object in the third sample video as the text label corresponding to the sample video.

[0311] It can be understood that if the third sample video acquired by the electronic device is a video that has been pre-annotated (labeled), the electronic device can also supplement the labels corresponding to each acquired third sample video according to the above steps a to b, and determine the micro-movements of the specified object in each third sample video.

[0312] In this way, the third sample videos can be annotated through the pre-trained video understanding model to obtain the labels corresponding to each third sample video. Compared with the manual labeling method, the efficiency of labeling can be improved and the cost of labeling sample videos can be reduced.

[0313] With respect to the above steps 2 and 3, after obtaining each third sample video and the text label corresponding to the third sample video, the electronic device can input each third sample video into the visual feature extraction network in the multimodal large model of the initial structure to obtain the video feature of the third sample video. And, input the text label corresponding to each third sample video into the text feature extraction network in the multimodal large model of the initial structure to obtain the text feature of the text label corresponding to the third sample video.

[0314] In one implementation, the multimodal large model of the initial structure is pre-trained based on fourth sample videos and text labels corresponding to the fourth sample videos; the text label corresponding to each fourth sample video is used to describe the fourth sample video.

[0315] The scene involved in any fourth sample video is not limited to the scene currently requiring micro-motion recognition. For example, the fourth sample video may be a video in a public data set that has been annotated in advance.

[0316] The visual feature extraction network in the multimodal large model of the initial structure can also be called a visual encoder. The visual encoder can be: CLIP ViT-Large (Contrastive Language-Image Pre-training Vision Transformer –Large, Contrastive Language-Image Pre-training and Visual Transformer Network –Large). CLIP ViT-Large is composed of 24 stacked Transformer Encoders.

[0317] Accordingly, for each sample video, the video features of the sample video can be obtained based on the visual encoder. That is, for each video frame in a sample video, the video frame can be input into the input layer of the visual encoder, and the output layer of the visual encoder can output the feature vector of the video frame. Furthermore, the mean of the feature vectors of all video frames of the sample video can be calculated as the video feature of the video.

[0318] The text feature extraction network in the multimodal large model of the initial structure can also be called a text encoder (Textencoder). The text encoder can be a BERT (Bidirectional Encoder Representations from Transformers) model. The text encoder is composed of 12 stacked TransformerEncoders (Transformer network encoders).

[0319] Correspondingly, for the text label corresponding to each sample video, the text features of the text label corresponding to the sample video can be obtained based on the text encoder. That is, BERT first looks up the input text label, converts each word in the text label into a one-dimensional vector as the model input, and inputs the one-dimensional vector into the Encoder, and further extracts the features after adding the position code to obtain the text features of the text label corresponding to the sample video.

[0320] Accordingly, in the pre-training process of the multimodal large model of the initial structure, after obtaining the video features of each fourth sample video and the text features of the text label corresponding to the fourth sample video, the electronic device can obtain a loss value based on the difference between the video features of each fourth sample video and the text features of the text label corresponding to the fourth sample video; based on the obtained loss value, the model parameters of the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached. In this way, a pre-trained multimodal large model of the initial structure is obtained.

[0321] For the above step 4, after obtaining the video features of each third sample video and the text label corresponding to the third sample video through the multimodal large model of the initial structure, the loss value can be obtained based on the difference between the video features of each third sample video and the text features of the text label corresponding to the third sample video. Then, based on the obtained loss value, the model parameters of the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached.

[0322] In one implementation, the above step five includes: based on the obtained loss value, adjusting the model parameters of the specified network layer of the visual feature extraction network in the multimodal large model of the initial structure until the preset convergence conditions are reached.

[0323] Among them, among the network layers included in the visual feature extraction network in the multimodal large model of the initial structure, the specified network layer is located after other network layers in order; when adjusting the model parameters of the specified network layer, the model parameters of the text feature extraction network in the multimodal large model of the initial structure are kept unchanged.

[0324] In the embodiment of the present application, since the multimodal large model of the initial structure is pre-trained based on the fourth sample video and the text label corresponding to the fourth sample video, that is, the multimodal large model of the initial structure can better learn the text features of the text.

[0325] It is understandable that the differences between the collected video contents in different scenes are usually large. Therefore, in order to enable the multimodal large model to better learn the scene where micro-motion recognition is currently required, only the model parameters of the visual feature extraction network can be adjusted so that the visual feature extraction network can extract video features suitable for the current scene. At the same time, in order to maintain the original performance of the visual feature extraction network, only the model parameters of some network layers (i.e., specified network layers) in the visual feature extraction network can be adjusted.

[0326] For example, the following table is a better parameter adjustment specification table determined by technicians based on multiple trainings, as shown in Table 1:

[0327] Table 1

[0328]

[0329] Among them, the parameter represents the training parameter in the training process of the multimodal large model. The value of the number of trainable layers of the video encoder is the last 4 layers, which means that the electronic device can adjust the model parameters of the last 4 network layers in the visual feature extraction network in the multimodal large model of the initial structure. The number of trainable layers of the text encoder is 0, which means that when adjusting the model parameters of the specified network layer, the model parameters of the text feature extraction network in the multimodal large model of the initial structure remain unchanged. The learning rate and weight decay coefficient of the multimodal large model are pre-set by the technician. For example, the learning rate can be 1e-5, and the weight decay coefficient can be 0.01. The video sampling frame rate is 8, which means: for all video frames within one second in the sample video, 8 frames can be sampled for processing.

[0330] Based on the above processing, due to the differences between the collected video content in different scenes, it is usually large. Therefore, in order to enable the multimodal large model to better learn the current scene that requires micro-motion recognition, only the model parameters of the visual feature extraction network can be adjusted so that the visual feature extraction network can extract video features suitable for the current scene. At the same time, in order to maintain the original performance of the visual feature extraction network, only the model parameters of some network layers (ie, specified network layers) in the visual feature extraction network can be adjusted. This can reduce the training cost of the multimodal large model and ensure that the trained multimodal large model can accurately learn the relationship between the video and video features of the current micro-motion recognition scene.

[0331] In some embodiments, the text feature extraction network in the micro-motion recognition model is a text feature extraction network in a pre-trained multimodal large model.

[0332] In an embodiment of the present application, when constructing a micro-motion recognition model, the text feature extraction network in the pre-trained multimodal large model can be used as the text feature extraction network in the micro-motion recognition model. Subsequently, in the process of training the micro-motion recognition model, the model parameters of the text feature extraction network can be kept unchanged. Among them, the training process of the micro-motion recognition model will be described in subsequent embodiments.

[0333] In this way, since the multimodal large model of the initial structure can better learn the text features of the text, the text feature extraction network in the pre-trained multimodal large model is used as the text feature extraction network in the micro-motion recognition model. Later, in the process of training the micro-motion recognition model, the model parameters of the text feature extraction network can be kept unchanged. In this way, the training cost of the micro-motion recognition model can be reduced.

[0334] In addition, since the text feature extraction network in the micro-motion recognition model is the text feature extraction network in the multimodal large model, and the initial structure of the multimodal large model is pre-trained based on the video (the fourth sample video) in the public data set and the text labels corresponding to the video. Therefore, the micro-motion recognition model can also maintain good recognition capabilities when processing videos of other scenes, which can improve the generalization ability of the micro-motion recognition model. And when processing videos shot with different video quality, lighting conditions, and viewing angle changes, it has strong robustness.

[0335] In some embodiments, see Figure 6 , Figure 6 A flowchart of a micro-motion recognition model training method provided in an embodiment of the present application. The micro-motion recognition model training process includes the following steps:

[0336] S601: Obtain a first sample video and a text label corresponding to the first sample video.

[0337] S602: Input the text label corresponding to each first sample video into the text feature extraction network in the micro-motion recognition model of the initial structure to obtain the text feature of the text label corresponding to the first sample video.

[0338] S603: For each first sample video, input the first sample video into a visual feature extraction network in a micro-motion recognition model of an initial structure to obtain a first sample video feature corresponding to the first sample video.

[0339] Among them, the first sample video feature includes multiple first sample feature vectors, the multiple first sample feature vectors correspond one-to-one to the video frames contained in the first sample video, and the position sequence of the multiple first sample feature vectors in the first sample video feature is consistent with the timing of the corresponding video frames in the first sample video.

[0340] S604: Calculate the adjustment amount corresponding to each first sample feature vector according to the position sequence of the first sample feature vector in the first sample video feature.

[0341] S605: Using each adjustment amount, adjust the first sample feature vector corresponding to the first sample video feature to obtain a second sample video feature corresponding to the first sample video.

[0342] S606: For each first sample video, a loss value is obtained based on a difference between a feature of a second sample video corresponding to the first sample video and a text feature of a text label corresponding to the first sample video.

[0343] S607: Based on the obtained loss value, the model parameters of the visual feature extraction network in the micro-motion recognition model of the initial structure are adjusted until a preset convergence condition is reached.

[0344] In the embodiment of the present application, with respect to step S601, the scene involved in any first sample video is consistent with the scene currently requiring micro-motion recognition.

[0345] Taking the above driving scenario as an example, the first sample video may be: a video collected when the driver triggers a fatigue driving event during historical driving, or a video collected when the driver does not trigger a fatigue driving event during historical driving.

[0346] Accordingly, for each first sample video, the electronic device can obtain the text label corresponding to each first sample video. The text label corresponding to each first sample video indicates: the micro-motion (including positive micro-motion and negative micro-motion) of the specified object in the first sample video. The method of obtaining the text label corresponding to each first sample video can refer to the process of obtaining the text label corresponding to each third sample video according to step a to step b in the above embodiment, which will not be described in detail here.

[0347] With respect to step S602, after obtaining each first sample video and the text label corresponding to the first sample video, the electronic device can input the text label corresponding to each first sample video into the text feature extraction network in the micro-motion recognition model of the initial structure to obtain the text feature of the text label corresponding to the first sample video. The text feature extraction network in the micro-motion recognition model of the initial structure can be a text feature extraction network in a pre-trained multimodal large model.

[0348] With respect to step S603 , for each first sample video, the electronic device may input the first sample video into a visual feature extraction network in a micro-motion recognition model of an initial structure to obtain a first sample video feature corresponding to the first sample video.

[0349] The process of obtaining the first sample video features corresponding to each first sample video may refer to the description related to obtaining the first video features to be identified of the video to be identified in the above step S102, and will not be elaborated here.

[0350] For step S604, the adjustment amount corresponding to each first sample feature vector is calculated according to the position sequence of each first feature vector in the first sample video feature. Reference may be made to the description of calculating the adjustment amount corresponding to the first feature vector to be identified according to the position sequence of each first feature vector to be identified in the first video feature to be identified in step S103 above, which will not be described in detail here.

[0351] For step S605, the first sample feature vector corresponding to the first sample video feature is adjusted by using each adjustment amount to obtain the second sample video feature corresponding to the first sample video. The description related to adjusting the first feature vector corresponding to the first video feature to be identified by using each adjustment amount to obtain the second video feature to be identified in the above step S104 can be referred to, and will not be repeated here.

[0352] For steps S606 to S607, for each first sample video, the electronic device can obtain a loss value based on the difference between the second sample video feature corresponding to the first sample video and the text feature of the text label corresponding to the first sample video. Based on the obtained loss value, the model parameters of the visual feature extraction network in the micro-motion recognition model of the initial structure are adjusted until a preset convergence condition is reached.

[0353] In some embodiments, S605 includes:

[0354] S6051: using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain an adjusted first sample video feature;

[0355] S6052: performing convolution processing on the adjusted first sample video features to obtain multiple feature maps with the same dimensions;

[0356] S6053: For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain a second sample feature vector corresponding to the position;

[0357] S6054: Input each obtained second sample feature vector into the sequence network in the micro-motion recognition model, and combine the feature vectors output by the sequence network at multiple time points to obtain the second sample video features corresponding to the first sample video.

[0358] In the embodiment of the present application, for steps S6051 to S6054, reference may be made to the descriptions related to steps S1041 to S1044 in the above embodiment, and no further details are given here.

[0359] Based on the above processing, after obtaining the adjusted first sample video features, the electronic device can perform convolution processing on the adjusted first sample video features to extract deep features of the sample video. Furthermore, the feature maps with multiple dimensions obtained by convolution can be transformed to obtain the second sample feature vector corresponding to each position in the feature map, and each second sample feature vector is input into the sequence network, and the feature vectors output by the sequence network at multiple times are combined to obtain the second sample video features. It can be seen from the characteristics of the sequence network that the sequence network is suitable for sequence data of various lengths, which can ensure the stability of processing the second sample feature vectors corresponding to different sample videos, and thus, can improve the scope of application of the micro-motion recognition method provided in the present application.

[0360] See also Figure 7 , Figure 7 A schematic diagram of performing micro-motion recognition provided in an embodiment of the present application. Figure 7 The production model in contains a lightweight visual encoder (i.e., the visual feature extraction network in the micro-motion recognition model in this application).

[0361] First, the electronic device can use the video stream (i.e., the third sample video in this application) and the multiple text labels corresponding to the video stream (i.e., the text labels corresponding to the third sample video in this application) to train a multimodal large model. Among them, the multimodal large model includes a visual encoder (i.e., a visual feature extraction network) and a text encoder (i.e., a text feature extraction network). Correspondingly, for each video stream, the video features of the video stream can be obtained by the visual encoder, and the text features of each text label of the video stream can be obtained by the text encoder. Furthermore, the electronic device can perform similarity calculation, that is, calculate the similarities between the video features of the video stream and the text features of each text label of the video stream (i.e., Sim1=0.9, Sim2=0.2, ...). Correspondingly, the electronic device can use the similarities between the video features of the video stream and the text features of each text label of the video stream to determine the difference between the video features of the video stream and the text features of each text label of the video stream, and obtain the loss value; based on the obtained loss value, the model parameters of the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached.

[0362] Accordingly, after the training of the multimodal large model is completed, the text encoder in the multimodal large model can be used to extract features of each preset prompt word to obtain the text features of each preset prompt word. After the electronic device obtains the video to be identified, it can input each video frame in the video to be identified into the visual feature extraction network (i.e., lightweight visual encoder) in the micro-motion recognition model to obtain the first video feature to be identified, and obtain the second video feature to be identified of the video to be identified according to the above steps S103 to S104. Furthermore, the electronic device can perform similarity calculation, that is, calculate the similarity between the second video feature to be identified of the video to be identified and the text features of each preset prompt word (i.e., Sim1=0.8, Sim2=0.1, ...). Accordingly, it can be determined that the micro-motion represented by the preset prompt word with a corresponding similarity greater than the first threshold exists in the specified object in the video to be identified.

[0363] See also Figure 8 , Figure 8 A structural diagram of a micro-motion recognition model provided in an embodiment of the present application. The micro-motion recognition model includes a backbone network part and a sequence network part. Among them, the backbone network can be the CNN in the above embodiment, and the sequence network can be the RNN in the above embodiment. Accordingly, after obtaining a video stream, the electronic device can extract features of the video stream through the backbone network to obtain a first video feature to be identified (i.e., a high-dimensional feature). Furthermore, the first feature vector corresponding to the first video feature to be identified can be adjusted to obtain the adjusted first video feature to be identified, and the adjusted first video feature to be identified can be convoluted to obtain multiple feature maps with the same dimensions; for each position in the obtained feature map, the value of the element at the position on each feature map is obtained to obtain the second feature vector to be identified corresponding to the position; each second feature vector to be identified is input into the sequence network, and the feature vector output by the sequence network at multiple times is combined to obtain the second video feature to be identified. Furthermore, based on the matching result of the second video feature to be identified and the text feature of the preset prompt word (i.e., performing a sequence similarity measurement), it is determined whether the specified object in the video to be identified has a micro-motion represented by the preset prompt word.

[0364] See also Fig. 9 , Fig. 9 A schematic diagram of a process for calculating similarity provided in an embodiment of the present application. Fig. 9In the figure, 0, 1, 2...n respectively represent a video frame in the video to be identified, 0 represents the first video frame in the video to be identified, 1 represents the second video frame in the video to be identified, 2 represents the third video frame in the video to be identified, and n represents the n+1th video frame in the video to be identified. The part corresponding to the dotted box represents: the feature vector of the nth video frame in the video to be identified (that is, the first feature vector to be identified in this application), and the encoding result of the adjustment amount corresponding to the first feature vector to be identified. That is, Frame+Position Embedding (encoding result of video frame and position). time represents the encoding result of each video frame and position based on the timing of each video frame in the video to be identified. Correspondingly, for each video frame in the video to be identified, the encoding result of the video frame and position can be obtained, that is, in this application, each adjustment amount is used to adjust the first feature vector to be identified corresponding to the first video feature to be identified. Accordingly, the electronic device can obtain each second feature vector to be identified according to the above steps S1042 to S1043, and input each second feature vector to be identified into the sequence network (i.e., LSTM), and obtain the second video feature to be identified according to step S1044. Furthermore, the electronic device can calculate the similarity between the second video feature to be identified and the text feature of each preset prompt word.

[0365] In the technical solution of this application, the operations involved in obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0366] It should be noted that the video containing the specified object in this embodiment comes from a public dataset.

[0367] Based on the same inventive concept, the present application embodiment provides a micro-motion recognition device. Fig.10 , Fig.10 A structural diagram of a micro-motion recognition device provided in an embodiment of the present application, the device comprising:

[0368] The to-be-recognized video acquisition module 1001 is used to acquire the to-be-recognized video containing the specified object;

[0369] The visual feature extraction module 1002 is used to input each video frame in the video to be identified into the visual feature extraction network in the pre-trained micro-motion recognition model to obtain a first video feature to be identified; wherein the first video feature to be identified includes a plurality of first feature vectors to be identified, the plurality of first feature vectors to be identified correspond one-to-one to the video frames included in the video to be identified, and the position sequence of each first feature vector to be identified in the first video feature to be identified is consistent with the time sequence of the corresponding video frame in the video to be identified;

[0370] The adjustment amount calculation module 1003 is used to calculate the adjustment amount corresponding to each first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified;

[0371] An adjustment module 1004 is used to adjust the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount to obtain a second video feature to be identified;

[0372] The matching module 1005 is used to determine whether the specified object in the video to be identified has the micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word; wherein the text feature of the preset prompt word is obtained by extracting the feature of the preset prompt word based on the text feature extraction network in the micro-motion recognition model; the micro-motion recognition model is trained based on the first sample video and the text label corresponding to the first sample video, and each text label corresponding to the first sample video represents the micro-motion existing in the specified object in the first sample video.

[0373] In some embodiments, the adjustment amount calculation module 1003 is specifically used to:

[0374] According to the position sequence of each first feature vector to be identified in the first video feature to be identified, the adjustment amount corresponding to the first feature vector to be identified is calculated according to the following formula:

[0375] ;

[0376] The value range of i is between 1 and ; Indicates An adjustment amount corresponding to a first feature vector to be identified; represents the total number of video frames contained in the video to be identified; d represents the dimension of the first feature vector to be identified.

[0377] In some embodiments, the micro-motion recognition model further comprises a sequence network;

[0378] The adjustment module 1004 includes:

[0379] A first adjustment submodule, configured to adjust the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount, to obtain an adjusted first video feature to be identified;

[0380] A first convolution submodule is used to perform convolution processing on the adjusted first video feature to be identified to obtain multiple feature maps with the same dimension;

[0381] A second feature vector acquisition submodule to be identified is used to obtain the value of the element at each position in the obtained feature map, and obtain the second feature vector to be identified corresponding to the position;

[0382] The second to-be-recognized video feature acquisition submodule is used to input each obtained second to-be-recognized feature vector into the sequence network, and combine the feature vectors output by the sequence network at multiple moments to obtain the second to-be-recognized video feature.

[0383] In some embodiments, the sequence network comprises a hidden layer and an output layer;

[0384] The second to-be-identified video feature acquisition submodule is specifically used to:

[0385] Using the weight of the sequence network at the current moment, a weighted sum of the second feature vectors to be identified is calculated as the weighted feature vector at the current moment; wherein the weight of the sequence network at the current moment is obtained based on the second feature vectors to be identified and the feature vector output by the hidden layer at the previous moment;

[0386] Inputting the feature vector output by the output layer at the last moment, the weighted feature vector at the current moment, and the feature vector output by the hidden layer at the last moment into the hidden layer, to obtain the feature vector output by the output layer at the current moment, and the feature vector output by the hidden layer at the current moment;

[0387] Returning to the step of using the weights of the sequence network at the current moment to calculate the weighted sum of the second feature vectors to be identified as the weighted feature vector at the current moment, until a specified number of feature vectors output by the output layer are obtained;

[0388] The specified number of feature vectors output by the output layer are combined to obtain the second video features to be identified.

[0389] In some embodiments, the preset prompt words include preset positive prompt words, and a positive prompt word indicates a positive micro-motion existing in a specified object when a specified event is triggered;

[0390] The device also includes:

[0391] A designated event determination module is used to determine whether the designated object in the video to be identified has a micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word. If the designated object in the video to be identified has the positive example micro-motion, then determine that the designated object in the video to be identified triggers the designated event.

[0392] In some embodiments, the preset prompt word includes a preset negative prompt word, and the negative prompt word represents a negative example micro-motion other than the positive example micro-motion;

[0393] The designated event determination module is specifically used to:

[0394] If the specified object in the video to be identified has the positive micro-motion and does not have the negative micro-motion, it is determined that the specified object in the video to be identified triggers the specified event.

[0395] In some embodiments, the preset prompt word is obtained based on the following steps:

[0396] Obtain each candidate prompt word and a second sample video; wherein the candidate prompt word includes: a plurality of candidate positive prompt words and a plurality of candidate negative prompt words; the second sample video includes a plurality of second positive sample videos and a plurality of second negative sample videos, a specified object in any second positive sample video triggers the specified event, and a specified object in any second negative sample video does not trigger the specified event;

[0397] Input each candidate prompt word into a text feature extraction network in a pre-trained multimodal large model to obtain text features of each candidate prompt word;

[0398] Inputting each second sample video into a visual feature extraction network in a pre-trained multimodal large model to obtain video features of each second sample video;

[0399] For each second sample video, calculating the similarity between the text features of each candidate prompt word and the video features of the second sample video;

[0400] Based on the obtained similarity, a prompt word that matches the second positive sample video and does not match the second negative sample video is determined from each candidate positive prompt word as a preset positive prompt word; and a prompt word that does not match the second positive sample video and matches the second negative sample video is determined from each candidate negative prompt word as a preset negative prompt word.

[0401] In some embodiments, the text feature extraction network in the micro-motion recognition model is a text feature extraction network in a pre-trained multimodal large model.

[0402] In some embodiments, the training process of the multimodal large model includes the following steps:

[0403] Obtaining a third sample video and a text label corresponding to the third sample video; wherein the text label corresponding to each third sample video indicates a micro-motion of a specified object in the third sample video;

[0404] Input each third sample video into the visual feature extraction network in the multimodal large model of the initial structure to obtain the video features of the third sample video;

[0405] Inputting the text label corresponding to each third sample video into the text feature extraction network in the multimodal large model of the initial structure to obtain the text feature of the text label corresponding to the third sample video;

[0406] Obtaining a loss value based on a difference between a video feature of each third sample video and a text feature of a text label corresponding to the third sample video;

[0407] Based on the obtained loss value, the model parameters of the multimodal large model of the initial structure are adjusted until the preset convergence conditions are reached.

[0408] In some embodiments, the training process of the micro-motion recognition model includes the following steps:

[0409] Obtaining a first sample video and a text label corresponding to the first sample video;

[0410] Inputting the text label corresponding to each first sample video into the text feature extraction network in the micro-motion recognition model of the initial structure to obtain the text feature of the text label corresponding to the first sample video;

[0411] For each first sample video, the first sample video is input into a visual feature extraction network in a micro-motion recognition model of an initial structure to obtain a first sample video feature corresponding to the first sample video; wherein the first sample video feature includes a plurality of first sample feature vectors, the plurality of first sample feature vectors correspond one-to-one to video frames included in the first sample video, and the position sequence of the plurality of first sample feature vectors in the first sample video feature is consistent with the time sequence of the corresponding video frames in the first sample video;

[0412] Calculate the adjustment amount corresponding to each first sample feature vector according to the position sequence of the first sample feature vector in the first sample video feature;

[0413] Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain a second sample video feature corresponding to the first sample video;

[0414] For each first sample video, a loss value is obtained based on a difference between a feature of a second sample video corresponding to the first sample video and a text feature of a text label corresponding to the first sample video;

[0415] Based on the obtained loss value, the model parameters of the visual feature extraction network in the micro-motion recognition model of the initial structure are adjusted until the preset convergence conditions are reached.

[0416] In some embodiments, using each adjustment amount to adjust the first sample feature vector corresponding to the first sample video feature to obtain the second sample video feature corresponding to the first sample video includes:

[0417] Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain an adjusted first sample video feature;

[0418] Performing convolution processing on the adjusted first sample video features to obtain multiple feature maps with the same dimensions;

[0419] For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain the second sample feature vector corresponding to the position;

[0420] The obtained second sample feature vectors are input into the sequence network in the micro-motion recognition model, and the feature vectors output by the sequence network at multiple moments are combined to obtain the second sample video features corresponding to the first sample video.

[0421] In some embodiments, the process of obtaining the text label corresponding to the sample video includes the following steps:

[0422] For each sample video, the sample video and the labeled prompt word are input into a pre-trained video understanding model to obtain a result output by the video understanding model; wherein the labeled prompt word indicates that the video understanding model determines the micro-motion of a specified object in the input sample video;

[0423] Based on the results output by the video understanding model, a text label corresponding to the sample video is obtained.

[0424] In some embodiments, the multimodal large model of the initial structure is obtained by pre-training based on the fourth sample video and the text label corresponding to the fourth sample video; the text label corresponding to each fourth sample video is used to describe the fourth sample video;

[0425] The method of adjusting the model parameters of the multimodal large model of the initial structure based on the obtained loss value until a preset convergence condition is reached includes:

[0426] Based on the obtained loss value, the model parameters of the specified network layer of the visual feature extraction network in the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached;

[0427] Among them, among the network layers included in the visual feature extraction network in the multimodal large model of the initial structure, the specified network layer is located after other network layers in order; when adjusting the model parameters of the specified network layer, the model parameters of the text feature extraction network in the multimodal large model of the initial structure are kept unchanged.

[0428] In some embodiments, the matching module 1005 is specifically used to:

[0429] Calculating the similarity between the second video feature to be identified and the text feature of the preset prompt word;

[0430] If the calculated similarity is greater than a first threshold, it is determined that the specified object in the video to be identified has a micro-motion represented by the preset prompt word;

[0431] If the calculated similarity is not greater than the first threshold, it is determined that the specified object in the to-be-identified video does not have the micro-motion represented by the preset prompt word.

[0432] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0433] Memory 1101, used for storing computer programs;

[0434] The processor 1102 is used to implement the steps of any of the above-mentioned micro-motion recognition methods when executing the program stored in the memory 1101.

[0435] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0436] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0437] The communication interface is used for communication between the above electronic device and other devices.

[0438] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0439] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0440] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned micro-motion recognition methods are implemented.

[0441] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the micro-motion recognition methods in the above embodiments.

[0442] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0443] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0444] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0445] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A micro-motion recognition method, characterized in that: The method comprises: Get the video to be identified that contains the specified object; Input each video frame in the video to be identified into a visual feature extraction network in a pre-trained micro-motion recognition model to obtain a first video feature to be identified; wherein the first video feature to be identified includes a plurality of first feature vectors to be identified, the plurality of first feature vectors to be identified correspond one-to-one to the video frames included in the video to be identified, and the position sequence of each first feature vector to be identified in the first video feature to be identified is consistent with the time sequence of the corresponding video frame in the video to be identified; Calculating the adjustment amount corresponding to each first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified; Using each adjustment amount, adjusting the first feature vector to be identified corresponding to the first video feature to be identified, to obtain a second video feature to be identified; Based on the matching result of the second to-be-recognized video feature and the text feature of the preset prompt word, determining whether the specified object in the to-be-recognized video has a micro-motion represented by the preset prompt word; Among them, the text features of the preset prompt words are obtained by extracting features of the preset prompt words based on the text feature extraction network in the micro-motion recognition model; the micro-motion recognition model is trained based on the first sample video and the text labels corresponding to the first sample video, and the text labels corresponding to each first sample video represent the micro-motions of the specified object in the first sample video.

2. The method according to claim 1, characterized in that The step of calculating the adjustment amount corresponding to each first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified includes: According to the position sequence of each first feature vector to be identified in the first video feature to be identified, the adjustment amount corresponding to the first feature vector to be identified is calculated according to the following formula: ； The value range of i is between 1 and ; Indicates An adjustment amount corresponding to a first feature vector to be identified; represents the total number of video frames contained in the video to be identified; d represents the dimension of the first feature vector to be identified.

3. The method according to claim 1, characterized in that The micro-motion recognition model also includes a sequence network; The step of adjusting the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount to obtain the second video feature to be identified includes: Using each adjustment amount, adjusting the first feature vector to be identified corresponding to the first video feature to be identified, to obtain an adjusted first video feature to be identified; Performing convolution processing on the adjusted first video feature to be identified to obtain multiple feature maps with the same dimensions; For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain the second feature vector to be identified corresponding to the position; The obtained second feature vectors to be identified are input into the sequence network, and the feature vectors output by the sequence network at multiple moments are combined to obtain the second video features to be identified.

4. The method according to claim 3, characterized in that The sequence network comprises a hidden layer and an output layer; The step of inputting each of the obtained second feature vectors to be identified into the sequence network, and combining the feature vectors output by the sequence network at multiple moments to obtain the second video features to be identified includes: Using the weight of the sequence network at the current moment, a weighted sum of the second feature vectors to be identified is calculated as the weighted feature vector at the current moment; wherein the weight of the sequence network at the current moment is obtained based on the second feature vectors to be identified and the feature vector output by the hidden layer at the previous moment; Inputting the feature vector output by the output layer at the last moment, the weighted feature vector at the current moment, and the feature vector output by the hidden layer at the last moment into the hidden layer, to obtain the feature vector output by the output layer at the current moment, and the feature vector output by the hidden layer at the current moment; Returning to the step of using the weights of the sequence network at the current moment to calculate the weighted sum of the second feature vectors to be identified as the weighted feature vector at the current moment, until a specified number of feature vectors output by the output layer are obtained; The specified number of feature vectors output by the output layer are combined to obtain the second video features to be identified.

5. The method according to claim 1, characterized in that The preset prompt words include preset positive prompt words, and a positive prompt word indicates a positive micro-motion existing in the specified object when the specified event is triggered; After determining whether the specified object in the video to be identified has a micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word, the method further includes: If the specified object in the video to be identified has the positive micro-motion, then it is determined that the specified object in the video to be identified triggers the specified event.

6. The method according to claim 5, characterized in that The preset prompt words include preset negative prompt words, and the negative prompt words represent negative micro-movements other than the positive micro-movements; If the specified object in the video to be identified has the positive micro-motion, determining that the specified object in the video to be identified triggers the specified event includes: If the specified object in the video to be identified has the positive micro-motion and does not have the negative micro-motion, it is determined that the specified object in the video to be identified triggers the specified event.

7. The method according to claim 6, characterized in that The preset prompt word is obtained based on the following steps: Obtain each candidate prompt word and a second sample video; wherein the candidate prompt word includes: a plurality of candidate positive prompt words and a plurality of candidate negative prompt words; the second sample video includes a plurality of second positive sample videos and a plurality of second negative sample videos, a specified object in any second positive sample video triggers the specified event, and a specified object in any second negative sample video does not trigger the specified event; Input each candidate prompt word into a text feature extraction network in a pre-trained multimodal large model to obtain text features of each candidate prompt word; Inputting each second sample video into a visual feature extraction network in a pre-trained multimodal large model to obtain video features of each second sample video; For each second sample video, calculating the similarity between the text features of each candidate prompt word and the video features of the second sample video; Based on the obtained similarity, a prompt word that matches the second positive sample video and does not match the second negative sample video is determined from each candidate positive prompt word as a preset positive prompt word; and a prompt word that does not match the second positive sample video and matches the second negative sample video is determined from each candidate negative prompt word as a preset negative prompt word.

8. The method according to claim 7, characterized in that The text feature extraction network in the micro-motion recognition model is a text feature extraction network in a pre-trained multimodal large model.

9. The method according to claim 7, characterized in that: The training process of the multimodal large model includes the following steps: Obtaining a third sample video and a text label corresponding to the third sample video; wherein the text label corresponding to each third sample video indicates a micro-motion of a specified object in the third sample video; Input each third sample video into the visual feature extraction network in the multimodal large model of the initial structure to obtain the video features of the third sample video; Inputting the text label corresponding to each third sample video into the text feature extraction network in the multimodal large model of the initial structure to obtain the text feature of the text label corresponding to the third sample video; Obtaining a loss value based on a difference between a video feature of each third sample video and a text feature of a text label corresponding to the third sample video; Based on the obtained loss value, the model parameters of the multimodal large model of the initial structure are adjusted until the preset convergence conditions are reached.

10. The method according to claim 1, characterized in that The training process of the micro-motion recognition model includes the following steps: Obtaining a first sample video and a text label corresponding to the first sample video; Inputting the text label corresponding to each first sample video into the text feature extraction network in the micro-motion recognition model of the initial structure to obtain the text feature of the text label corresponding to the first sample video; For each first sample video, the first sample video is input into a visual feature extraction network in a micro-motion recognition model of an initial structure to obtain a first sample video feature corresponding to the first sample video; wherein the first sample video feature includes a plurality of first sample feature vectors, the plurality of first sample feature vectors correspond one-to-one to video frames included in the first sample video, and the position sequence of the plurality of first sample feature vectors in the first sample video feature is consistent with the time sequence of the corresponding video frames in the first sample video; Calculate the adjustment amount corresponding to each first sample feature vector according to the position sequence of the first sample feature vector in the first sample video feature; Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain a second sample video feature corresponding to the first sample video; For each first sample video, a loss value is obtained based on a difference between a feature of a second sample video corresponding to the first sample video and a text feature of a text label corresponding to the first sample video; Based on the obtained loss value, the model parameters of the visual feature extraction network in the micro-motion recognition model of the initial structure are adjusted until the preset convergence conditions are reached.

11. The method according to claim 10, characterized in that The method of adjusting the first sample feature vector corresponding to the first sample video feature by using each adjustment amount to obtain the second sample video feature corresponding to the first sample video includes: Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain an adjusted first sample video feature; Performing convolution processing on the adjusted first sample video features to obtain multiple feature maps with the same dimensions; For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain the second sample feature vector corresponding to the position; The obtained second sample feature vectors are input into the sequence network in the micro-motion recognition model, and the feature vectors output by the sequence network at multiple moments are combined to obtain the second sample video features corresponding to the first sample video.

12. The method according to claim 9 or 10, characterized in that The process of obtaining the text labels corresponding to the sample videos includes the following steps: For each sample video, the sample video and the labeled prompt word are input into a pre-trained video understanding model to obtain a result output by the video understanding model; wherein the labeled prompt word indicates that the video understanding model determines the micro-motion of a specified object in the input sample video; Based on the results output by the video understanding model, a text label corresponding to the sample video is obtained.

13. The method according to claim 9, characterized in that The multimodal large model of the initial structure is obtained by pre-training based on the fourth sample video and the text label corresponding to the fourth sample video; the text label corresponding to each fourth sample video is used to describe the fourth sample video; The method of adjusting the model parameters of the multimodal large model of the initial structure based on the obtained loss value until a preset convergence condition is reached includes: Based on the obtained loss value, the model parameters of the specified network layer of the visual feature extraction network in the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached; Among them, among the network layers included in the visual feature extraction network in the multimodal large model of the initial structure, the specified network layer is located after other network layers in order; when adjusting the model parameters of the specified network layer, the model parameters of the text feature extraction network in the multimodal large model of the initial structure are kept unchanged.

14. The method according to claim 1, characterized in that The determining whether the specified object in the video to be identified has a micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word comprises: Calculating the similarity between the second video feature to be identified and the text feature of the preset prompt word; If the calculated similarity is greater than a first threshold, it is determined that the specified object in the video to be identified has a micro-motion represented by the preset prompt word; If the calculated similarity is not greater than the first threshold, it is determined that the specified object in the to-be-identified video does not have the micro-motion represented by the preset prompt word.

15. A micro-motion recognition device, characterized in that: The device comprises: A module for acquiring videos to be identified, used to acquire videos to be identified that contain a specified object; A visual feature extraction module, used for inputting each video frame in the video to be identified into a visual feature extraction network in a pre-trained micro-motion recognition model to obtain a first video feature to be identified; wherein the first video feature to be identified includes a plurality of first feature vectors to be identified, the plurality of first feature vectors to be identified correspond one-to-one to the video frames included in the video to be identified, and the position sequence of each first feature vector to be identified in the first video feature to be identified is consistent with the time sequence of the corresponding video frame in the video to be identified; An adjustment amount calculation module, used for calculating the adjustment amount corresponding to each first feature vector to be identified according to the position sequence of the first feature vector to be identified in the first video feature to be identified; An adjustment module, configured to adjust the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount, so as to obtain a second video feature to be identified; A matching module is used to determine whether a specified object in the video to be identified has a micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word; wherein the text feature of the preset prompt word is obtained by extracting features of the preset prompt word based on the text feature extraction network in the micro-motion recognition model; the micro-motion recognition model is trained based on a first sample video and the text label corresponding to the first sample video, and each text label corresponding to the first sample video represents a micro-motion existing in the specified object in the first sample video.

16. The device according to claim 15, characterized in that The adjustment amount calculation module is specifically used for: According to the position sequence of each first feature vector to be identified in the first video feature to be identified, the adjustment amount corresponding to the first feature vector to be identified is calculated according to the following formula: ； The value range of i is between 1 and ; Indicates An adjustment amount corresponding to a first feature vector to be identified; represents the total number of video frames contained in the video to be identified; d represents the dimension of the first feature vector to be identified; and / or, The micro-motion recognition model also includes a sequence network; The adjustment module comprises: A first adjustment submodule, configured to adjust the first feature vector to be identified corresponding to the first video feature to be identified by using each adjustment amount, to obtain an adjusted first video feature to be identified; A first convolution submodule is used to perform convolution processing on the adjusted first video feature to be identified to obtain multiple feature maps with the same dimension; A second feature vector acquisition submodule to be identified is used to obtain the value of the element at each position in the obtained feature map, and obtain the second feature vector to be identified corresponding to the position; A second to-be-recognized video feature acquisition submodule, used for inputting each obtained second to-be-recognized feature vector into the sequence network, and combining the feature vectors output by the sequence network at multiple moments to obtain the second to-be-recognized video feature; and / or, The sequence network comprises a hidden layer and an output layer; The second to-be-identified video feature acquisition submodule is specifically used to: Using the weight of the sequence network at the current moment, a weighted sum of the second feature vectors to be identified is calculated as the weighted feature vector at the current moment; wherein the weight of the sequence network at the current moment is obtained based on the second feature vectors to be identified and the feature vector output by the hidden layer at the previous moment; Inputting the feature vector output by the output layer at the last moment, the weighted feature vector at the current moment, and the feature vector output by the hidden layer at the last moment into the hidden layer, to obtain the feature vector output by the output layer at the current moment, and the feature vector output by the hidden layer at the current moment; Returning to the step of using the weights of the sequence network at the current moment to calculate the weighted sum of the second feature vectors to be identified as the weighted feature vector at the current moment, until a specified number of feature vectors output by the output layer are obtained; Combining a specified number of feature vectors output by the output layer to obtain a second video feature to be identified; and / or, The preset prompt words include preset positive prompt words, and a positive prompt word indicates a positive micro-motion existing in the specified object when the specified event is triggered; The device also includes: A designated event determination module, configured to determine whether the designated object in the video to be identified has the micro-motion represented by the preset prompt word based on the matching result of the second video feature to be identified and the text feature of the preset prompt word, and if the designated object in the video to be identified has the positive micro-motion, determine that the designated object in the video to be identified triggers the designated event; and / or, The preset prompt words include preset negative prompt words, and the negative prompt words represent negative micro-movements other than the positive micro-movements; The designated event determination module is specifically used to: If the specified object in the video to be identified has the positive micro-motion and does not have the negative micro-motion, then determining that the specified object in the video to be identified triggers the specified event; and / or, The preset prompt word is obtained based on the following steps: Obtain each candidate prompt word and a second sample video; wherein the candidate prompt word includes: a plurality of candidate positive prompt words and a plurality of candidate negative prompt words; the second sample video includes a plurality of second positive sample videos and a plurality of second negative sample videos, a specified object in any second positive sample video triggers the specified event, and a specified object in any second negative sample video does not trigger the specified event; Input each candidate prompt word into a text feature extraction network in a pre-trained multimodal large model to obtain text features of each candidate prompt word; Inputting each second sample video into a visual feature extraction network in a pre-trained multimodal large model to obtain video features of each second sample video; For each second sample video, calculating the similarity between the text features of each candidate prompt word and the video features of the second sample video; Based on the obtained similarity, determine, from each candidate positive prompt word, a prompt word that matches the second positive sample video and does not match the second negative sample video as a preset positive prompt word; and determine, from each candidate negative prompt word, a prompt word that does not match the second positive sample video and matches the second negative sample video as a preset negative prompt word; and / or, The text feature extraction network in the micro-motion recognition model is a text feature extraction network in a pre-trained multimodal large model; and / or, The training process of the multimodal large model includes the following steps: Obtaining a third sample video and a text label corresponding to the third sample video; wherein the text label corresponding to each third sample video indicates a micro-motion of a specified object in the third sample video; Input each third sample video into the visual feature extraction network in the multimodal large model of the initial structure to obtain the video features of the third sample video; Inputting the text label corresponding to each third sample video into the text feature extraction network in the multimodal large model of the initial structure to obtain the text feature of the text label corresponding to the third sample video; Obtaining a loss value based on a difference between a video feature of each third sample video and a text feature of a text label corresponding to the third sample video; Based on the obtained loss value, the model parameters of the multimodal large model of the initial structure are adjusted until the preset convergence conditions are reached; and / or, The training process of the micro-motion recognition model includes the following steps: Obtaining a first sample video and a text label corresponding to the first sample video; Inputting the text label corresponding to each first sample video into the text feature extraction network in the micro-motion recognition model of the initial structure to obtain the text feature of the text label corresponding to the first sample video; For each first sample video, the first sample video is input into a visual feature extraction network in a micro-motion recognition model of an initial structure to obtain a first sample video feature corresponding to the first sample video; wherein the first sample video feature includes a plurality of first sample feature vectors, the plurality of first sample feature vectors correspond one-to-one to video frames included in the first sample video, and the position sequence of the plurality of first sample feature vectors in the first sample video feature is consistent with the time sequence of the corresponding video frames in the first sample video; Calculate the adjustment amount corresponding to each first sample feature vector according to the position sequence of the first sample feature vector in the first sample video feature; Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain a second sample video feature corresponding to the first sample video; For each first sample video, a loss value is obtained based on a difference between a feature of a second sample video corresponding to the first sample video and a text feature of a text label corresponding to the first sample video; Based on the obtained loss value, the model parameters of the visual feature extraction network in the micro-motion recognition model of the initial structure are adjusted until the preset convergence conditions are reached; and / or, The method of adjusting the first sample feature vector corresponding to the first sample video feature by using each adjustment amount to obtain the second sample video feature corresponding to the first sample video includes: Using each adjustment amount, adjusting the first sample feature vector corresponding to the first sample video feature to obtain an adjusted first sample video feature; Performing convolution processing on the adjusted first sample video features to obtain multiple feature maps with the same dimensions; For each position in the obtained feature map, obtain the value of the element at the position on each feature map to obtain the second sample feature vector corresponding to the position; Inputting each obtained second sample feature vector into the sequence network in the micro-motion recognition model, and combining the feature vectors output by the sequence network at multiple moments to obtain the second sample video feature corresponding to the first sample video; and / or, The process of obtaining the text label corresponding to the sample video includes the following steps: For each sample video, the sample video and the labeled prompt word are input into a pre-trained video understanding model to obtain a result output by the video understanding model; wherein the labeled prompt word indicates that the video understanding model determines the micro-motion of a specified object in the input sample video; Based on the result output by the video understanding large model, obtain a text label corresponding to the sample video; and / or, The multimodal large model of the initial structure is obtained by pre-training based on the fourth sample video and the text label corresponding to the fourth sample video; the text label corresponding to each fourth sample video is used to describe the fourth sample video; The method of adjusting the model parameters of the multimodal large model of the initial structure based on the obtained loss value until a preset convergence condition is reached includes: Based on the obtained loss value, the model parameters of the specified network layer of the visual feature extraction network in the multimodal large model of the initial structure are adjusted until the preset convergence condition is reached; Among the network layers included in the visual feature extraction network in the multimodal large model of the initial structure, the specified network layer is located after the other network layers in order; when adjusting the model parameters of the specified network layer, the model parameters of the text feature extraction network in the multimodal large model of the initial structure are kept unchanged; And / or, the matching module is specifically used for: Calculating the similarity between the second video feature to be identified and the text feature of the preset prompt word; If the calculated similarity is greater than a first threshold, it is determined that the specified object in the video to be identified has a micro-motion represented by the preset prompt word; If the calculated similarity is not greater than the first threshold, it is determined that the specified object in the to-be-identified video does not have the micro-motion represented by the preset prompt word.

17. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-14 when executing a program stored in a memory.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images