Method, device and electronic equipment for detecting role behavior

Abstract

The embodiment of the application provides a role behavior detection method and device and electronic equipment, and relates to the technical field of data processing, which comprises the following steps: obtaining a to-be-detected video segment containing a role; based on the image features of the video frames in the to-be-detected video segment and auxiliary information, processing by using a pre-trained behavior detection model to obtain the role behavior detection result of the to-be-detected video segment; wherein the auxiliary information represents the audio features obtained by performing feature extraction on the audio frames in the to-be-detected video segment, and / or the objects contained in the to-be-detected video segment. Through the scheme, the detection result of the role behavior in the to-be-detected video segment can be determined by using multiple dimensions of information, so that the accuracy of determining the detection result of the role behavior in the video segment can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, and electronic device for detecting role behavior. Background Technology

[0002] To detect character behavior in video clips, related technologies involve inputting video frames from the video clip into a pre-trained behavior recognition model, such as the ViVIT (A Video Vision Transformer) model or the Uniformer (Unifified Transformer) model, to obtain the output of the behavior recognition model as the detection result of the character behavior. The video clip can be a recorded video, a film or television drama, or an animation; the character can be a person in the video clip. In the case of an animation or science fiction film, the character can also be a virtual non-human character, such as an anthropomorphic animal.

[0003] However, for some scenarios with similar behaviors, behavior recognition models have difficulty making accurate identifications. For example, a behavior recognition model may confuse a scenario where a character is drinking water with a scenario where a character is drinking alcohol.

[0004] It is evident that the accuracy of detection results for determining the behavior of characters in video clips still needs improvement in related technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, and electronic device for detecting character behavior, so as to improve the accuracy of detecting character behavior in video clips. The specific technical solution is as follows:

[0006] In a first aspect of this invention, a method for detecting role behavior is provided, the method comprising:

[0007] Obtain the video clip to be detected that contains the character;

[0008] Based on the image features and auxiliary information of the video frames in the video segment to be detected, a pre-trained behavior detection model is used for processing to obtain the role behavior detection result of the video segment to be detected; wherein, the auxiliary information represents: audio features obtained by feature extraction of audio frames in the video segment to be detected, and / or, the objects contained in the video segment to be detected.

[0009] Optionally, the step of processing the image features and auxiliary information of the video frames in the video segment to be detected using a pre-trained behavior detection model to obtain the role behavior detection result of the video segment to be detected includes:

[0010] The image features of video frames in the video segment to be detected are input into the pre-trained first behavior.

[0011] The detection model obtains a first intermediate detection result representing the behavior of a character in the video segment to be detected; wherein, the first behavior detection model is trained based on video frames of sample video segments containing characters and labels representing the behavior of characters in the sample video segments;

[0012] Based on the first intermediate detection result and the auxiliary detection result, the character behavior detection result of the video segment to be detected is determined;

[0013] The auxiliary detection results include: a second intermediate detection result and / or a third intermediate detection result; the second intermediate detection result is obtained by inputting audio features extracted from audio frames in the video segment to be detected into a pre-trained audio classification model, resulting in a representation of the character behavior in the video segment to be detected; the audio classification model is trained based on sample audio frames and labels representing the character behavior of the sample audio frames; the third intermediate detection result is obtained by inputting video frames in the video segment to be detected into a pre-trained target detection model, resulting in a representation of the objects contained in the video segment to be detected; the target detection model is trained based on video frames in the sample video segment and labels representing the objects contained in the sample video frames.

[0014] Optionally, determining the character behavior detection result of the video segment to be detected based on the first intermediate detection result and the auxiliary detection result includes:

[0015] If the character behavior represented by the first intermediate detection result belongs to the first type of behavior, the character behavior detection result of the video segment to be detected is determined based on the first intermediate detection result and the second intermediate detection result; wherein, the first type of behavior refers to the behavior of making a sound;

[0016] If the role behavior represented by the first intermediate detection result belongs to the second type of behavior, the role behavior detection result of the video segment to be detected is determined based on the first intermediate detection result and the third intermediate detection result; wherein, the second type of behavior refers to the behavior that requires the participation of a specified object.

[0017] Optionally, the step of processing the image features and auxiliary information of the video frames in the video segment to be detected using a pre-trained behavior detection model to obtain the role behavior detection result of the video segment to be detected includes:

[0018] The video frames and audio frames from the video segment to be detected are input into the feature extraction layer of a pre-trained second behavior detection model to obtain image features and audio features. The second behavior detection model is trained based on video frames and audio frames of sample video segments containing characters, and labels representing the character's behavior in the sample video segments. The second behavior detection model further includes: a fusion layer for fusing the input image features and audio features to obtain fused features; and a classification layer for obtaining an output representing the character's behavior based on the input fused features.

[0019] The obtained image features and audio features are input into the fusion layer to obtain the fused features to be utilized;

[0020] The fusion features to be utilized are input into the classification layer to obtain a fourth intermediate detection result representing the behavior of the characters in the video segment to be detected;

[0021] Based on the fourth intermediate detection result, the character behavior detection result of the video segment to be detected is determined.

[0022] Optionally, determining the character behavior detection result of the video segment to be detected based on the fourth intermediate detection result includes:

[0023] Determine whether the role behavior represented by the fourth intermediate detection result belongs to the second type of behavior; wherein, the second type of behavior refers to behavior that requires the participation of a specified object;

[0024] If so, based on the fourth intermediate detection result and the third intermediate detection result, the character behavior detection result of the video segment to be detected is determined; wherein, the third intermediate detection result is: the result representing the object contained in the video segment to be detected is obtained by inputting the video frames in the video segment to be detected into a pre-trained target detection model;

[0025] If not, the fourth intermediate detection result is determined as the character behavior detection result of the video segment to be detected.

[0026] In a second aspect of the invention, a device for detecting role behavior is also provided, the device comprising:

[0027] The video clip acquisition module is used to acquire video clips containing characters to be detected;

[0028] The processing module is used to process the image features and auxiliary information of the video frames in the video segment to be detected using a pre-trained behavior detection model to obtain the role behavior detection result of the video segment to be detected; wherein, the auxiliary information represents: audio features obtained by feature extraction of audio frames in the video segment to be detected, and / or, the objects contained in the video segment to be detected.

[0029] Optionally, the processing module includes:

[0030] The detection submodule is used to input the image features of the video frames in the video segment to be detected into a pre-trained first behavior detection model to obtain a first intermediate detection result representing the behavior of the character in the video segment to be detected; wherein, the first behavior detection model is trained based on the video frames of the sample video segment containing the character and the labels representing the behavior of the character in the sample video segment.

[0031] The first detection result determination submodule is used to determine the character behavior detection result of the video segment to be detected based on the first intermediate detection result and the auxiliary detection result;

[0032] The auxiliary detection results include: a second intermediate detection result and / or a third intermediate detection result; the second intermediate detection result is obtained by inputting audio features extracted from audio frames in the video segment to be detected into a pre-trained audio classification model, resulting in a representation of the character behavior in the video segment to be detected; the audio classification model is trained based on sample audio frames and labels representing the character behavior of the sample audio frames; the third intermediate detection result is obtained by inputting video frames in the video segment to be detected into a pre-trained target detection model, resulting in a representation of the objects contained in the video segment to be detected; the target detection model is trained based on video frames in the sample video segment and labels representing the objects contained in the sample video frames.

[0033] Optionally, the processing module includes:

[0034] The feature extraction submodule is used to input video frames and audio frames from the video segment to be detected into the feature extraction layer of a pre-trained second behavior detection model to obtain image features and audio features. The second behavior detection model is trained based on video frames and audio frames of sample video segments containing characters, and labels representing the character's behavior in the sample video segments. The second behavior detection model further includes: a fusion layer for fusing the input image features and audio features to obtain fused features; and a classification layer for obtaining an output representing the character's behavior based on the input fused features.

[0035] The feature fusion submodule is used to input the obtained image features and audio features into the fusion layer to obtain the fused features to be utilized;

[0036] The classification submodule is used to input the fusion features to be utilized into the classification layer to obtain a fourth intermediate detection result representing the behavior of the characters in the video segment to be detected;

[0037] The second detection result determination submodule is used to determine the character behavior detection result of the video segment to be detected based on the fourth intermediate detection result.

[0038] In a third aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0039] Memory, used to store computer programs;

[0040] A processor, when executing a program stored in memory, implements any of the above-described methods for detecting role behavior.

[0041] In a fourth aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the detection method for any of the aforementioned role behaviors.

[0042] In a fifth aspect of the invention, a computer program product comprising instructions is also provided, which, when run on a computer, causes the computer to perform any of the aforementioned character behaviors for detection.

[0043] This invention provides a method for detecting character behavior. The method involves acquiring a video segment containing a character; processing the image features and auxiliary information of the video frames within the video segment using a pre-trained behavior detection model to obtain the character behavior detection result of the video segment; wherein the auxiliary information represents: audio features obtained by feature extraction from audio frames in the video segment, and / or the objects contained in the video segment. Therefore, this solution utilizes a pre-trained behavior detection model to comprehensively process audio features and / or the objects contained in the video segment, as well as the image features of the video frames. This allows for the determination of the character behavior detection result in the video segment using multi-dimensional information, thereby improving the accuracy of the character behavior detection result. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0045] Figure 1 This is a flowchart of a method for detecting character behavior in an embodiment of the present invention;

[0046] Figure 2 This is another flowchart of the method for detecting character behavior in an embodiment of the present invention;

[0047] Figure 3 This is another flowchart of a method for detecting character behavior in an embodiment of the present invention;

[0048] Figure 4 This is another flowchart of a method for detecting character behavior in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of the character behavior detection device in an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0052] To improve the accuracy of detecting character behavior in video clips, this invention provides a method, apparatus, and electronic device for character behavior detection. This method can be applied to electronic devices such as smartphones, servers, and computers. Furthermore, this method can be implemented through a combination of software and hardware.

[0053] The method for detecting role behavior provided in this embodiment of the invention may include the following steps:

[0054] Obtain the video clip to be detected that contains the character;

[0055] Based on the image features and auxiliary information of the video frames in the video segment to be detected, a pre-trained behavior detection model is used to process the data and obtain the character behavior detection result of the video segment to be detected. The auxiliary information refers to the audio features obtained by feature extraction of the audio frames in the video segment to be detected, and / or the objects contained in the video segment to be detected.

[0056] In this embodiment, a video segment containing a character is acquired for detection. Based on the image features and auxiliary information of the video frames in the video segment, a pre-trained behavior detection model is used for processing to obtain the character behavior detection result of the video segment. The auxiliary information refers to: audio features obtained by feature extraction from the audio frames in the video segment, and / or the objects contained in the video segment. Therefore, this solution utilizes a pre-trained behavior detection model to process audio features and / or the objects contained in the video segment, as well as the image features of the video frames. This allows for the determination of the character behavior detection result in the video segment using multi-dimensional information, thereby improving the accuracy of the detection result.

[0057] The method for detecting character behavior provided in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] like Figure 1 As shown, the method for detecting character behavior provided in this embodiment of the invention may include the following steps:

[0059] S101, Obtain the video segment to be detected that contains the character;

[0060] The video segment to be detected can be any video segment containing characters that requires character behavior detection. For example, for movies and TV series uploaded to video platforms, to facilitate viewers' understanding of the video content, character behavior detection can be performed on video segments containing characters to obtain character behavior detection results. Then, when playing video segments containing characters, the corresponding character behavior detection results can be displayed to help viewers understand the video content. In this embodiment of the invention, the characters can be human figures. However, in cases where the video segment is in the form of animation, science fiction, etc., the characters can also be non-human figures, such as anthropomorphic animals.

[0061] To obtain the video segment to be detected, a video clip containing characters can be extracted from the movie video, TV series video, etc.

[0062] S102, based on the image features and auxiliary information of the video frames in the video segment to be detected, the pre-trained behavior detection model is used for processing to obtain the role behavior detection result of the video segment to be detected; wherein, the auxiliary information represents: the audio features obtained by feature extraction of the audio frames in the video segment to be detected, and / or, the objects contained in the video segment to be detected.

[0063] In one implementation, the behavior detection model can be a first behavior detection model, which can be a model with the ability to recognize character behavior in video, such as ViVIT (A Video Vision Transformer), Uniformer (Unifified Transformer), or InternVideo (a general video foundation model). The first behavior detection model can be trained based on video frames of sample video clips containing characters, and labels representing the character behavior in the sample video clips. The labels representing the character behavior in the sample video clips can be predefined based on the actual character behavior in the sample video clips.

[0064] The first behavior detection model may include a feature extraction layer. The image features of the video frames in the video segment to be detected can be obtained by extracting features from the video frames using the feature extraction layer in the first behavior detection model. The extracted image features can then be input into the subsequent network structure of the first behavior detection model for processing to obtain the role behavior detection result.

[0065] like Figure 2 As shown, in this implementation, the image features of the video frames in the video segment to be detected can be input into the pre-trained first behavior detection model to obtain the first intermediate detection result representing the behavior of the character in the video segment to be detected. Then, based on the first intermediate detection result and the auxiliary detection result, the behavior detection result of the video segment to be detected can be determined.

[0066] The auxiliary detection results may include: a second intermediate detection result and / or a third intermediate detection result; the second intermediate detection result is obtained by inputting the audio features extracted from the audio frames in the video segment to be detected into a pre-trained audio classification model, resulting in a representation of the character behavior in the video segment to be detected; the audio classification model is trained based on sample audio frames and the labels representing the character behavior of the sample audio frames; the third intermediate detection result is obtained by inputting the video frames in the video segment to be detected into a pre-trained object detection model, resulting in a representation of the objects contained in the video segment to be detected; the object detection model is trained based on the video frames of the sample video segment and the labels used to represent the objects contained in the sample video frames.

[0067] The aforementioned audio classification model can be any model capable of classifying audio, such as EcapaTdnn (a voiceprint recognition model), PANNs (Pretrained Audio Neural Networks), TDNN (Time-Delay Neural Network), Res2Net (a residual network), etc. The audio features of audio frames in the video segment to be detected can also be extracted using the feature extraction layer of this audio classification model.

[0068] The aforementioned object detection model can be any model capable of detecting objects in an image, such as Faster R-CNN (Faster Region-Convolutional Neural Network), YOLOv8 (an object detection algorithm), etc., used to identify objects contained in the input image.

[0069] In this implementation, while obtaining the first intermediate detection result based on the first behavior detection model, the audio frames in the video segment to be detected are input into a pre-trained audio classification model to obtain the second intermediate detection result, and / or the video frames in the video segment to be detected are input into a pre-trained target detection model to obtain the third intermediate detection result. Then, information fusion is performed on the obtained detection results to determine the role behavior detection result of the video segment to be detected.

[0070] The auxiliary detection result can be either the second intermediate detection result or the third intermediate detection result, or it can include both the second intermediate detection result and the third intermediate detection result.

[0071] In one implementation, after obtaining the first intermediate detection result and the auxiliary detection result, the decision to obtain the second or third intermediate detection result is based on the type of role behavior represented by the first intermediate detection result. For example, if the role behavior represented by the first intermediate detection result belongs to the first type of behavior, the role behavior detection result of the video segment to be detected can be determined based on the first and second intermediate detection results. Here, the first type of behavior refers to the behavior of making sounds; for example, the first type of behavior can include behaviors such as singing and giving speeches. In this case, since the first intermediate detection result is related to sound, the second intermediate detection result obtained based on the audio classification model can be used to assist in determining the role behavior detection result of the video segment to be detected, thereby improving the accuracy of determining the role behavior detection result.

[0072] If the role behavior represented by the first intermediate detection result belongs to the second type of behavior, the role behavior detection result of the video segment to be detected can be determined based on the first and third intermediate detection results. The second type of behavior refers to behaviors that require the participation of a specified object. For example, the second type of behavior could be actions such as drinking alcohol or drinking water, all of which require the participation of a specified object. The specified object can be pre-defined for each type of second behavior; for example, drinking alcohol requires the participation of a wine bottle, and drinking water requires the participation of a water bottle. In this case, the third intermediate detection result obtained based on the object detection model can be used to assist in determining the role behavior detection result of the video segment to be detected, thereby improving the accuracy of the determined role behavior detection result.

[0073] Understandably, if the role behavior represented by the first intermediate detection result belongs to the first type of behavior, the role behavior detection result of the video segment to be detected can also be determined based on the first intermediate detection result, the second intermediate detection result, and the third intermediate detection result simultaneously. Similarly, if the role behavior represented by the first intermediate detection result belongs to the second type of behavior, the role behavior detection result of the video segment to be detected can also be determined based on the first intermediate detection result, the second intermediate detection result, and the third intermediate detection result simultaneously, so as to make the role behavior detection result more accurate.

[0074] In one implementation, after obtaining the first intermediate detection result and the auxiliary detection result, the role behavior detection result of the video segment to be detected can be determined according to the pre-set decision tree.

[0075] For example, the process can be as follows Figure 3 As shown, when the first intermediate detection result indicates that the character's behavior is drinking, since drinking is similar to drinking water and can be easily confused, a decision tree can be pre-set for drinking behavior. The decision process represented by this decision tree can be as follows:

[0076] First, obtain the third intermediate detection result determined by the object detection model. If the third intermediate detection result indicates that there is a wine bottle in the video segment to be detected, then the character behavior detection result of the video segment to be detected is determined to be drinking behavior; if the third intermediate detection result indicates that there is a water bottle in the video segment to be detected, then the character behavior detection result of the video segment to be detected is determined to be drinking behavior.

[0077] If the first intermediate detection result indicates that the role behavior is a speech behavior, since speech behavior and singing behavior are easily confused, a decision tree can be pre-set for the speech behavior. The decision process represented by this decision tree can be as follows:

[0078] First, a second intermediate detection result determined based on an audio classification model is obtained. If the role behavior indicated by the second intermediate detection result is a speech behavior, then the role behavior detection result of the video segment to be detected is determined to be a speech behavior; if the role behavior indicated by the second intermediate detection result is a singing behavior, then the role behavior detection result of the video segment to be detected is determined to be a singing behavior. Further, if the role behavior indicated by the second intermediate detection result is another behavior, that is, neither a singing behavior nor a speech behavior, a third intermediate detection result can be obtained. If the third intermediate detection result indicates that the video segment to be detected contains an instrument, then the role behavior detection result of the video segment to be detected is determined to be a singing behavior; if the third intermediate detection result indicates that the video segment to be detected does not contain an instrument, then the role behavior detection result of the video segment to be detected is determined to be a speech behavior.

[0079] In this embodiment, a video segment containing a character is acquired for detection. Based on the image features and auxiliary information of the video frames in the video segment, a pre-trained behavior detection model is used for processing to obtain the character behavior detection result of the video segment. The auxiliary information refers to: audio features obtained by feature extraction from the audio frames in the video segment, and / or the objects contained in the video segment. Therefore, this solution utilizes a pre-trained behavior detection model to process audio features and / or the objects contained in the video segment, as well as the image features of the video frames. This allows for the determination of the character behavior detection result in the video segment using multi-dimensional information, thereby improving the accuracy of the detection result.

[0080] In another embodiment of the present invention, the above-mentioned processing of image features and auxiliary information of video frames in the video segment to be detected using a pre-trained behavior detection model to obtain the character behavior detection result of the video segment to be detected may include the following steps:

[0081] Step A1: Input the video frames and audio frames in the video segment to be detected into the feature extraction layer of the pre-trained second behavior detection model to obtain image features and audio features;

[0082] The second behavior detection model is trained based on video frames and audio frames of sample video clips containing characters, as well as labels representing the behavior of characters in the sample video clips. The second behavior detection model also includes: a fusion layer, which is used to fuse the input image features and audio features to obtain fused features; and a classification layer, which is used to obtain the output result representing the behavior of characters based on the input fused features.

[0083] In one implementation, the second behavior detection model may contain only two backbone networks for feature extraction, one for extracting features from the input video frames and the other for extracting features from the input audio frames. The backbone network for extracting features from the input video frames can be any neural network capable of image feature extraction, such as a CNN (Convolutional Neural Network); the backbone network for extracting features from the input audio frames can be any neural network capable of audio feature extraction, such as a ResNet (Residual Network).

[0084] The labels used to represent character behaviors in sample video clips can be pre-defined based on the actual character behaviors in the sample video clips. When training the second behavior detection model, video frames and audio frames from the sample video clips can be input into the second behavior detection model to obtain the predicted values ​​representing character behaviors output by the model. Then, a loss value is calculated based on the obtained predicted values ​​and the labels representing character behaviors in the sample video clips. The model parameters of the second behavior detection model are then adjusted based on this loss value until the model converges, resulting in the trained second behavior detection model.

[0085] Step A2: Input the obtained image features and audio features into the fusion layer to obtain the fusion features to be utilized;

[0086] The fusion layer can be any structure with feature fusion capabilities, such as the Cross-Attention structure, the Transformer structure, etc.

[0087] Step A3: Input the fusion features to be utilized into the classification layer to obtain the fourth intermediate detection result used to represent the behavior of the characters in the video segment to be detected;

[0088] The classification layer mentioned above can be any classifier, such as the Softmax (normalized exponential function) classifier, etc.

[0089] Step A4: Based on the fourth intermediate detection result, determine the character behavior detection result of the video segment to be detected.

[0090] In one implementation, the obtained fourth intermediate detection result can be directly used as the character behavior detection result of the video segment to be detected.

[0091] In another implementation, to further improve the accuracy of the role behavior detection results, it is also possible to first determine whether the role behavior represented by the fourth intermediate detection result belongs to the second type of behavior; where the second type of behavior refers to the behavior that requires the participation of a specified object.

[0092] If so, based on the fourth intermediate detection result and the third intermediate detection result, determine the role behavior detection result of the video segment to be detected; wherein, the third intermediate detection result is: inputting the video frames in the video segment to be detected into the pre-trained target detection model, and obtaining the result representing the objects contained in the video segment to be detected;

[0093] If not, the fourth intermediate detection result will be determined as the character behavior detection result of the video segment to be detected.

[0094] In this embodiment, the character behavior detection result of the video segment to be detected can also be determined based on the fourth intermediate detection result and the third intermediate detection result according to the preset decision tree.

[0095] For example, when the role behavior represented by the fourth intermediate detection result is drinking behavior, it can be regarded as belonging to the second type of behavior, and then the third intermediate detection result can be obtained. If the third intermediate detection result indicates that there is a wine bottle among the objects contained in the video segment to be detected, the role behavior detection result of the video segment to be detected can be determined to be drinking behavior; if the third intermediate detection result indicates that there is a water bottle among the objects contained in the video segment to be detected, the role behavior detection result of the video segment to be detected can be determined to be drinking behavior.

[0096] If the role behavior represented by the fourth intermediate detection result is a speech behavior, it can be regarded as not belonging to the second type of behavior, and the role behavior detection result of the video segment to be detected can be directly determined as a speech behavior.

[0097] It can be seen that, as Figure 4 As shown, the second behavior detection model in this embodiment can extract image features and audio features from video clips, then fuse the obtained image features and audio features to obtain fused features to be used, and then use the classification layer to obtain the detection result of the character behavior. The video frames and audio frames of the video clip can be input into the second behavior detection model to directly obtain the detection result of the character behavior output by the second behavior detection model. That is, the second behavior detection model can realize end-to-end detection of character behavior based on video frames and audio frames of video clips. The various layers in the second behavior detection model, including the feature extraction layer, fusion layer and classification layer, can be trained simultaneously. Therefore, it can have good performance and can further improve the accuracy of the detection result of character behavior based on video frames and audio frames of video clips.

[0098] Simultaneously, object detection models can be used to perform object detection on video frames of the video segment to be detected, obtaining a third intermediate detection result. Then, information fusion can be performed on the fourth intermediate detection result obtained based on fusion features and the third intermediate detection result to determine the final role behavior detection result. This enables the comprehensive utilization of three types of information: image features, audio features, and the objects contained in the video segment to be detected, thereby further improving the accuracy of the detection result for determining role behavior.

[0099] In one implementation, a third-party behavior detection model can be used to achieve end-to-end determination of character behavior detection results based on a combination of image classification, audio classification, and object detection information. The specific process is as follows:

[0100] Step B1: Using the feature extraction layer in the pre-trained third behavior detection model, extract the image features of the video frames in the video segment to be detected, and extract the audio features of the audio frames in the video segment to be detected; wherein, the third behavior detection model is trained based on the video frames and audio frames of the sample video segment containing the character, as well as the labels used to represent the character's behavior in the sample video segment and the labels used to represent the objects contained in the sample video frame.

[0101] Step B2: Using the image classification layer in the third behavior detection model, based on image features, obtain the image classification results of the character behavior represented by the video frames in the video segment to be detected;

[0102] Step B3: Using the audio classification layer in the third behavior detection model, based on audio features, obtain the audio classification results of the character behavior represented by the audio frames in the video segment to be detected.

[0103] Step B4: Using the target detection layer in the third line detection model, based on image features, obtain the object detection results representing the objects contained in the video segment to be detected;

[0104] Step B5: Based on the image classification results, audio classification results, and object detection results, determine the character behavior detection results of the video segment to be detected.

[0105] In this implementation, image classification, audio classification, and object detection can share a single backbone network for feature extraction. Then, image classification, audio classification, and object detection layers are used separately to obtain image classification results, audio classification results, and object detection results. Finally, based on a pre-defined decision tree, the character behavior detection result for the video segment to be detected is determined. Therefore, this third behavior detection model not only achieves end-to-end character behavior detection based on video and audio frames of a video segment, but also comprehensively utilizes image classification results, audio classification results, and object detection results to determine the character behavior detection result, further improving the model's performance.

[0106] When training the third behavior detection model, the video frames and audio frames of the sample video clip can be input into the third behavior detection model to obtain the first predicted value output by the image classification layer, the second predicted value output by the audio classification layer, and the third predicted value output by the object detection layer. Then, using the first predicted value, the second predicted value, the third predicted value, the labels used to represent the behavior of the characters in the sample video clip, and the labels used to represent the objects contained in the sample video frame, the loss value is calculated. Based on the obtained loss value, the model parameters of the third behavior detection model are adjusted until the model converges.

[0107] In this embodiment, the accuracy of detecting character behavior in video segments can be improved. Furthermore, by inputting video frames and audio frames from the video segment to be detected into the feature extraction layer of a pre-trained second behavior detection model, image features and audio features are obtained. The second behavior detection model further includes: a fusion layer for fusing the input image and audio features to obtain fused features; a classification layer for obtaining an output representing character behavior based on the input fused features; the obtained image and audio features are then input into the fusion layer to obtain fused features to be utilized; and the fused features to be utilized are input into the classification layer to obtain a fourth intermediate detection result representing character behavior in the video segment to be detected. Based on the fourth intermediate detection result, the character behavior detection result of the video segment to be detected is determined. This achieves end-to-end character behavior detection based on video frames and audio frames of video segments, improving model performance and further enhancing the accuracy of character behavior detection based on video frames and audio frames of video segments.

[0108] Based on the same inventive concept, embodiments of the present invention also provide a device for detecting role behavior, such as... Figure 5 As shown, the device includes:

[0109] The video clip acquisition module 501 is used to acquire video clips to be detected that contain characters;

[0110] The processing module 502 is used to process the image features and auxiliary information of the video frames in the video segment to be detected using a pre-trained behavior detection model to obtain the role behavior detection result of the video segment to be detected; wherein, the auxiliary information represents: audio features obtained by feature extraction of audio frames in the video segment to be detected, and / or, the objects contained in the video segment to be detected.

[0111] Optionally, the processing module 502 includes:

[0112] The detection submodule is used to input the image features of the video frames in the video segment to be detected into a pre-trained first behavior detection model to obtain a first intermediate detection result representing the behavior of the character in the video segment to be detected; wherein, the first behavior detection model is trained based on the video frames of the sample video segment containing the character and the labels representing the behavior of the character in the sample video segment.

[0113] The first detection result determination submodule is used to determine the character behavior detection result of the video segment to be detected based on the first intermediate detection result and the auxiliary detection result;

[0114] The auxiliary detection results include: a second intermediate detection result and / or a third intermediate detection result; the second intermediate detection result is obtained by inputting audio features extracted from audio frames in the video segment to be detected into a pre-trained audio classification model, resulting in a representation of the character behavior in the video segment to be detected; the audio classification model is trained based on sample audio frames and labels representing the character behavior of the sample audio frames; the third intermediate detection result is obtained by inputting video frames in the video segment to be detected into a pre-trained target detection model, resulting in a representation of the objects contained in the video segment to be detected; the target detection model is trained based on video frames in the sample video segment and labels representing the objects contained in the sample video frames.

[0115] Optionally, the first detection result determination submodule is specifically used to determine the character behavior detection result of the video segment to be detected based on the first intermediate detection result and the second intermediate detection result when the character behavior represented by the first intermediate detection result belongs to the first type of behavior; wherein, the first type of behavior refers to the behavior of making a sound;

[0116] If the role behavior represented by the first intermediate detection result belongs to the second type of behavior, the role behavior detection result of the video segment to be detected is determined based on the first intermediate detection result and the third intermediate detection result; wherein, the second type of behavior refers to the behavior that requires the participation of a specified object.

[0117] Optionally, the processing module 502 includes:

[0118] The feature extraction submodule is used to input video frames and audio frames from the video segment to be detected into the feature extraction layer of a pre-trained second behavior detection model to obtain image features and audio features. The second behavior detection model is trained based on video frames and audio frames of sample video segments containing characters, and labels representing the character's behavior in the sample video segments. The second behavior detection model further includes: a fusion layer for fusing the input image features and audio features to obtain fused features; and a classification layer for obtaining an output representing the character's behavior based on the input fused features.

[0119] The feature fusion submodule is used to input the obtained image features and audio features into the fusion layer to obtain the fused features to be utilized;

[0120] The classification submodule is used to input the fusion features to be utilized into the classification layer to obtain a fourth intermediate detection result representing the behavior of the characters in the video segment to be detected;

[0121] The second detection result determination submodule is used to determine the character behavior detection result of the video segment to be detected based on the fourth intermediate detection result.

[0122] Optionally, the second detection result determination submodule is specifically used for:

[0123] Determine whether the role behavior represented by the fourth intermediate detection result belongs to the second type of behavior; wherein, the second type of behavior refers to behavior that requires the participation of a specified object;

[0124] If so, based on the fourth intermediate detection result and the third intermediate detection result, the character behavior detection result of the video segment to be detected is determined; wherein, the third intermediate detection result is: the result representing the object contained in the video segment to be detected is obtained by inputting the video frames in the video segment to be detected into a pre-trained target detection model;

[0125] If not, the fourth intermediate detection result is determined as the character behavior detection result of the video segment to be detected.

[0126] This invention also provides an electronic device, such as... Figure 6As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0127] Memory 603 is used to store computer programs;

[0128] When the processor 601 executes the program stored in the memory 603, it implements the method for detecting the role behavior described in any of the above embodiments.

[0129] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0130] The communication interface is used for communication between the aforementioned terminal and other devices.

[0131] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0132] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0133] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the method for detecting role behavior as described in any of the above embodiments.

[0134] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the detection method for role behavior described in any of the above embodiments.

[0135] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0138] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for detecting character behavior, characterized in that, The method includes: Obtain the video clip to be detected that contains the character; Based on the image features and auxiliary information of the video frames in the video segment to be detected, a pre-trained behavior detection model is used to process the data to obtain the character behavior detection result of the video segment to be detected. The process, based on image features and auxiliary information from video frames in the video segment to be detected, utilizes a pre-trained behavior detection model to obtain the character behavior detection result for the video segment to be detected, including: The image features of the video frames in the video segment to be detected are input into a pre-trained first behavior detection model to obtain a first intermediate detection result representing the behavior of the character in the video segment to be detected; wherein, the first behavior detection model is trained based on the video frames of the sample video segment containing the character and the labels representing the behavior of the character in the sample video segment. If the character behavior represented by the first intermediate detection result belongs to the first type of behavior, the auxiliary information includes audio features extracted from the audio frames in the video segment to be detected. Based on the first intermediate detection result and the second intermediate detection result, the character behavior detection result of the video segment to be detected is determined. The first type of behavior refers to the behavior of making a sound. The second intermediate detection result is obtained by inputting the audio features extracted from the audio frames in the video segment to be detected into a pre-trained audio classification model, resulting in a representation of the character behavior in the video segment to be detected. The audio classification model is trained based on sample audio frames and the labels of the character behaviors represented by the sample audio frames. When the role behavior represented by the first intermediate detection result belongs to the second type of behavior, the auxiliary information includes the objects contained in the video segment to be detected. Based on the first intermediate detection result and the third intermediate detection result, the role behavior detection result of the video segment to be detected is determined; wherein, the second type of behavior indicates behavior that requires the participation of a specified object; the third intermediate detection result is: the result representing the objects contained in the video segment to be detected obtained by inputting the video frames in the video segment to be detected into a pre-trained target detection model; the target detection model is: trained based on the video frames of the sample video segment and the labels used to represent the objects contained in the video frames of the sample video segment.

2. A method for detecting character behavior, characterized in that, The method includes: Obtain the video clip to be detected that contains the character; Based on the image features and auxiliary information of the video frames in the video segment to be detected, a pre-trained behavior detection model is used for processing to obtain the character behavior detection result of the video segment to be detected; wherein, the auxiliary information includes audio features obtained by feature extraction of audio frames in the video segment to be detected; The process, based on image features and auxiliary information from video frames in the video segment to be detected, utilizes a pre-trained behavior detection model to obtain the character behavior detection result for the video segment to be detected, including: The video frames and audio frames from the video segment to be detected are input into the feature extraction layer of a pre-trained second behavior detection model to obtain image features and audio features. The second behavior detection model is trained based on video frames and audio frames of sample video segments containing characters, and labels representing the character's behavior in the sample video segments. The second behavior detection model further includes: a fusion layer for fusing the input image features and audio features to obtain fused features; and a classification layer for obtaining an output representing the character's behavior based on the input fused features. The obtained image features and audio features are input into the fusion layer to obtain the fused features to be utilized; The fusion features to be utilized are input into the classification layer to obtain a fourth intermediate detection result representing the behavior of the characters in the video segment to be detected; Based on the fourth intermediate detection result, the character behavior detection result of the video segment to be detected is determined; The step of determining the character behavior detection result of the video segment to be detected based on the fourth intermediate detection result includes: Determine whether the role behavior represented by the fourth intermediate detection result belongs to the second type of behavior; wherein, the second type of behavior refers to behavior that requires the participation of a specified object; If so, the auxiliary information also includes the objects contained in the video segment to be detected. Based on the fourth intermediate detection result and the third intermediate detection result, the role behavior detection result of the video segment to be detected is determined. The third intermediate detection result is: the result representing the objects contained in the video segment to be detected is obtained by inputting the video frames in the video segment to be detected into a pre-trained target detection model. If not, the fourth intermediate detection result is determined as the character behavior detection result of the video segment to be detected.

3. A device for detecting character behavior, characterized in that, The device includes: The video clip acquisition module is used to acquire video clips containing characters to be detected; The processing module is used to process the image features and auxiliary information of the video frames in the video segment to be detected using a pre-trained behavior detection model to obtain the character behavior detection result of the video segment to be detected. The processing module includes: The detection submodule is used to input the image features of the video frames in the video segment to be detected into a pre-trained first behavior detection model to obtain a first intermediate detection result representing the behavior of the character in the video segment to be detected; wherein, the first behavior detection model is trained based on the video frames of the sample video segment containing the character and the labels representing the behavior of the character in the sample video segment. The first detection result determination submodule is used to determine the character behavior detection result of the video segment under test based on the first intermediate detection result and the second intermediate detection result when the character behavior represented by the first intermediate detection result belongs to the first type of behavior. The auxiliary information includes audio features extracted from audio frames in the video segment under test. If the character behavior represented by the first intermediate detection result belongs to the second type of behavior, the auxiliary information includes objects contained in the video segment under test. The first intermediate detection result and the third intermediate detection result are used to determine the character behavior detection result of the video segment under test. Wherein, the first type of behavior represents the act of emitting sound; the second type of behavior represents the act of needing to be directed. The first intermediate detection result is the behavior of the target object involved; the second intermediate detection result is the audio features obtained by extracting features from the audio frames in the video segment to be detected, inputting them into a pre-trained audio classification model to obtain a result representing the behavior of the character in the video segment to be detected; the audio classification model is trained based on sample audio frames and labels representing the behavior of the character represented by the sample audio frames; the third intermediate detection result is the video frames in the video segment to be detected input into a pre-trained target detection model to obtain a result representing the object contained in the video segment to be detected; the target detection model is trained based on the video frames of the sample video segment and labels representing the objects contained in the video frames of the sample video segment.

4. A device for detecting character behavior, characterized in that, The device includes: The video clip acquisition module is used to acquire video clips containing characters to be detected; The processing module is used to process the image features and auxiliary information of the video frames in the video segment to be detected using a pre-trained behavior detection model to obtain the character behavior detection result of the video segment to be detected; wherein, the auxiliary information includes audio features obtained by feature extraction of audio frames in the video segment to be detected. The processing module includes: The feature extraction submodule is used to input video frames and audio frames from the video segment to be detected into the feature extraction layer of a pre-trained second behavior detection model to obtain image features and audio features. The second behavior detection model is trained based on video frames and audio frames of sample video segments containing characters, and labels representing the character's behavior in the sample video segments. The second behavior detection model further includes: a fusion layer for fusing the input image features and audio features to obtain fused features; and a classification layer for obtaining an output representing the character's behavior based on the input fused features. The feature fusion submodule is used to input the obtained image features and audio features into the fusion layer to obtain the fused features to be utilized; The classification submodule is used to input the fusion features to be utilized into the classification layer to obtain a fourth intermediate detection result representing the behavior of the characters in the video segment to be detected; The second detection result determination submodule is used to determine the character behavior detection result of the video segment to be detected based on the fourth intermediate detection result; Specifically, based on the fourth intermediate detection result, the character behavior detection result of the video segment to be detected is determined, including: Determine whether the role behavior represented by the fourth intermediate detection result belongs to the second type of behavior; wherein, the second type of behavior refers to behavior that requires the participation of a specified object; If so, the auxiliary information also includes the objects contained in the video segment to be detected. Based on the fourth intermediate detection result and the third intermediate detection result, the role behavior detection result of the video segment to be detected is determined. The third intermediate detection result is: the result representing the objects contained in the video segment to be detected is obtained by inputting the video frames in the video segment to be detected into a pre-trained target detection model. If not, the fourth intermediate detection result is determined as the character behavior detection result of the video segment to be detected.

5. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-2.

Images