A multimedia content recognition method and system based on cross-modal feature fusion

Abstract

The application relates to the technical field of action recognition, in particular to a multimedia content recognition method and system based on cross-modal feature fusion, which comprises the following steps: acquiring to-be-recognized multimedia, extracting a video frame sequence and an audio sequence from the to-be-recognized multimedia, performing key frame recognition on the video frame sequence to obtain a key frame sequence, performing emotion analysis on the audio sequence to obtain a character emotion sequence set, performing character target region detection on the key frame sequence to obtain a plurality of key character region sets, performing character interaction feature extraction on the key frame sequence based on the plurality of key character region sets to obtain a character interaction feature vector, and performing content recognition on the to-be-recognized multimedia according to the character emotion sequence set and the character interaction feature vector to obtain a multimedia content recognition result. According to the application, the key video frame can be adaptively determined according to the character behavior change in the video content, and the information of multiple modes such as video, voice and text can be fused, so that the accuracy of violent behavior recognition is improved.

Description

Technical Field

[0001] This invention relates to the field of action recognition technology, and in particular to a multimedia content recognition method and system based on cross-modal feature fusion. Background Technology

[0002] With the development of internet video transmission technology, a large amount of multimedia video content has been generated in live streaming, short videos, and online video platforms. Since some video content may contain violent acts such as fighting, attacks, and conflicts, the automatic identification of violent behavior in video content is of great significance for online content supervision, abnormal behavior detection, and video security auditing.

[0003] In existing technologies, violent behavior recognition is usually achieved by using video image analysis. For example, by sampling video frames, using target detection, action recognition and other technologies to extract the human targets and action features in the video, and judging whether there is violent behavior in the video based on the action features.

[0004] However, existing methods for identifying violent behavior primarily rely on the movement features of individuals in video images. However, some real-world scenarios may appear very similar to violent acts, but are not actually violent, such as fights in movies or playful brawls between friends. This leads existing methods to easily misidentify non-violent behavior as violent. Furthermore, using fixed-interval sampling for detection can easily miss crucial behavioral changes such as the onset of an attack, approach of individuals, and physical contact, reducing accuracy. Therefore, a method is needed that can adaptively determine key video frames based on changes in human behavior within the video content and integrate information from multiple modalities, including video, audio, and text, to improve the accuracy of violent behavior identification. Summary of the Invention

[0005] This invention provides a multimedia content recognition method and a computer-readable storage medium based on cross-modal feature fusion. Its main purpose is to adaptively determine key video frames based on changes in human behavior in video content, and to fuse information from multiple modalities such as video, audio, and text to improve the accuracy of violent behavior recognition.

[0006] To achieve the above objectives, the present invention provides a multimedia content recognition method based on cross-modal feature fusion, comprising: Acquire the multimedia to be identified, and extract the video frame sequence and audio sequence from the multimedia to be identified; Perform keyframe recognition on the video frame sequence to obtain the keyframe sequence; Perform sentiment analysis on the audio sequences to obtain a set of character emotion sequences; Perform character target region detection on the keyframe sequence to obtain multiple key character region sets; Based on multiple key character regions, character interaction features are extracted from keyframe sequences to obtain character interaction feature vectors. Based on the character's emotional sequence set and the character's interaction feature vector, the multimedia to be identified is subjected to content recognition, and the multimedia content recognition result is obtained.

[0007] Optionally, performing keyframe recognition on the video frame sequence to obtain a keyframe sequence includes: For each video frame in the video frame sequence, human region recognition is performed to obtain multiple sets of human target regions, where each set of human target regions corresponds one-to-one with a video frame. A sliding window extraction operation is performed on the video frame sequence to obtain multiple video frame combinations, wherein the video frame combination includes two combined video frames; Extract one video frame combination from multiple video frame combinations sequentially to obtain the target video frame combination, and perform the following operations on the target video frame combination: Extract two target person region sets corresponding to the combination of target video frames from multiple target person region sets; Determine the difference region of the target person based on the two target person region sets; Extract two image regions of the target person based on the difference region of the target person; Calculate the grayscale difference between two images of a person's region to obtain the amount of change in the person's region; Keyframes are obtained by performing keyframe detection based on changes in the character's area. The keyframes are extracted and summarized in chronological order by using a sliding window that combines multiple video frames to obtain a keyframe sequence.

[0008] Optionally, calculating the grayscale difference between two person region images to obtain the change in the person region includes: Determine the foreground background region and the background region. Extract multiple grayscale values ​​of foreground background pixels from the foreground background region and extract multiple grayscale values ​​of background pixels from the background region. The grayscale values ​​of foreground background pixels and background background pixels correspond one-to-one. Multiple background grayscale values ​​are obtained by subtracting multiple front background pixel grayscale values ​​from multiple back background pixel grayscale values. Extract the background brightness change from multiple background grayscale changes; Calculate the image difference between the two images of the person's region to obtain the difference image; Extract multiple difference grayscale values ​​from the difference image, and calculate the average of the absolute values ​​of the multiple difference grayscale values ​​to obtain the amount of grayscale change of the person; The change in the figure's grayscale is obtained by subtracting the change in the background brightness.

[0009] Optionally, the step of performing sentiment analysis on the audio sequence to obtain a set of character emotion sequences includes: Perform source separation on the audio sequence to obtain multiple candidate speech source sequences; Perform voice association on multiple candidate sound source speech sequences to obtain multiple target voice sequences; Speech segmentation is performed on multiple target person speech sequences to obtain multiple speech segment sets, where each speech segment set corresponds one-to-one with a target person speech sequence and includes multiple speech segments. For each of the multiple speech segment sets, perform the following operation: Identify multiple multimodal emotion feature groups based on a set of speech segments; Emotional state recognition is performed on multiple multimodal emotion feature groups to obtain a sequence of human emotional states, wherein the sequence of human emotional states includes multiple human emotional states, and each human emotional state corresponds one-to-one with a speech segment. By summarizing the sequences of characters' emotional states, we obtain a set of character emotional sequences.

[0010] Optionally, the sound source separation of the audio sequence to obtain multiple candidate sound source speech sequences includes: Denoising is performed on the audio sequence to obtain the denoised audio sequence; Perform time-frequency transformation on the noise-reduced audio sequence to obtain a frequency domain audio sequence, and extract audio time-frequency features from the frequency domain audio sequence; The audio time-frequency features are decomposed to obtain multiple sound source spectral features; Speech reconstruction is performed on the spectral features of multiple sound sources to obtain multiple candidate sound source speech sequences.

[0011] Optionally, the step of performing person-to-person speech association on multiple candidate sound source speech sequences to obtain multiple target person speech sequences includes: Extract multiple mouth region sequences from multiple target regions of individuals; Cross-frame mouth motion analysis was performed on multiple mouth region sequences to obtain multiple mouth motion feature sequences; Multiple mouth movement time segments were determined based on multiple mouth movement feature sequences; Speech activity detection is performed on speech sequences from multiple candidate sound sources to obtain multiple sets of speech activity time periods; Based on multiple mouth movement time periods and multiple speech activity time periods, person association is performed on multiple candidate sound source speech sequences to obtain multiple target person speech sequences.

[0012] Optionally, the step of performing cross-frame mouth motion analysis on multiple mouth region sequences to obtain multiple mouth motion feature sequences includes: Extract a mouth region sequence from multiple mouth region sequences sequentially to obtain the target mouth region sequence, and then perform the following operations on the target mouth region sequence: Perform lip keypoint detection on each target mouth region in the target mouth region sequence to obtain multiple lip keypoint sets, where each lip keypoint set corresponds one-to-one with a target mouth region; Based on multiple lip keypoint sets, position normalization and scale normalization are performed on each target mouth region in the target mouth region sequence to obtain a normalized mouth region sequence. A sliding window extraction operation is performed on the normalized mouth region sequence to obtain multiple combinations of adjacent mouth regions; Extract one adjacent mouth region combination from multiple adjacent mouth region combinations sequentially to obtain the target adjacent mouth region combination, and perform the following operations on the target adjacent mouth region combination: Lip key point detection is performed on the combination of adjacent mouth regions of the target to obtain two target lip key point sets. Based on the two target lip key point sets, the lip opening and closing change characteristics, the relative displacement characteristics of lip key points, and the mouth shape change characteristics are calculated. Vector concatenation is performed on the lip opening and closing characteristics, the relative displacement characteristics of key lip points, and the mouth shape change characteristics to obtain mouth motion features; Based on the extraction of mouth motion features in a sliding window combination of multiple adjacent mouth regions, the mouth motion features are summarized in order from first to last to obtain the mouth motion feature sequence. By summarizing the mouth movement feature sequences, multiple mouth movement feature sequences are obtained.

[0013] Optionally, the step of identifying multiple multimodal emotion feature groups based on the speech segment set includes: Extract speech segments one by one from the speech segment set to obtain the target speech segment, and perform the following operations on the target speech segment: Extract the video frame subsequence corresponding to the target audio segment from the video frame sequence, and obtain the audio character region sequence of the video frame subsequence; Acoustic emotion feature extraction is performed on the target speech segment to obtain the target acoustic emotion features; Speech recognition is performed on the target speech segment to obtain the target text segment. Then, text emotion features are extracted from the target text segment to obtain the target text emotion features. Visual emotion features are extracted from the speech character region sequence to obtain the target visual emotion features; The target acoustic emotion features, target text emotion features, and target visual emotion features are denoted as the multimodal emotion feature group; By summarizing the multimodal emotion feature groups, multiple multimodal emotion feature groups are obtained.

[0014] Optionally, the step of extracting character interaction features from the keyframe sequence based on multiple key character region sets to obtain a character interaction feature vector includes: Perform cross-frame target association on multiple key character regions to obtain the trajectories of multiple character targets; Perform pairwise combinations on the target trajectories of multiple characters to obtain multiple character interaction pairs; Extract one character interaction pair from multiple character interaction pairs sequentially to obtain the target character interaction pair, and perform the following operations on the target character interaction pair: Extract the execution center position boundary of the target character's interaction to obtain two center position sequences and two character region boundary sequences; Calculate the characteristics of human motion changes and human approach changes based on two central position sequences; Calculate the contact change characteristics of the two figures based on the boundary sequences of their respective regions; Perform vector concatenation on the character's movement change features, character's approach change features, and character's contact change features to obtain interactive action change sub-vectors; The interaction action change sub-vectors are summarized to obtain multiple interaction action change sub-vectors. The multiple interaction action change sub-vectors are then concatenated to obtain the character interaction features.

[0015] To achieve the above objectives, the present invention also provides a multimedia content recognition system based on cross-modal feature fusion, comprising: The content acquisition module is used to acquire the multimedia to be identified and extract video frame sequences and audio sequences from the multimedia to be identified; The keyframe recognition module is used to perform keyframe recognition on the video frame sequence to obtain the keyframe sequence. The sentiment analysis module is used to perform sentiment analysis on audio sequences to obtain a set of character emotion sequences; The content recognition module is used to perform character target region detection on keyframe sequences to obtain multiple key character region sets. Based on the multiple key character region sets, it performs character interaction feature extraction on keyframe sequences to obtain character interaction feature vectors. Based on the character emotion sequence set and character interaction feature vectors, it performs content recognition on the multimedia to be recognized to obtain multimedia content recognition results.

[0016] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: Memory, storing at least one instruction; The processor executes the instructions stored in the memory to implement the multimedia content recognition method based on cross-modal feature fusion described above.

[0017] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the multimedia content recognition method based on cross-modal feature fusion described above.

[0018] To address the problems described in the background art, this invention acquires multimedia to be identified, extracts video frame sequences and audio sequences from the multimedia, and performs keyframe recognition on the video frame sequences to obtain a keyframe sequence. Since violent behavior typically occurs only during a portion of the multimedia content, and existing technologies usually select video frames from the video frame sequence at fixed time intervals, key video frames reflecting the change in violent behavior, such as the start of an attack, the approach of a person, and contact with a person, are easily missed. Therefore, this invention performs change analysis on the target area of ​​the person in the video frame sequence, and determines keyframes based on the changes in the person area, enabling subsequent recognition processes to analyze video content that reflects changes in the person's actions. Simultaneously, since video acquisition is easily affected by factors such as changes in lighting, camera shake, and background object movement, pixel changes between adjacent video frames cannot accurately represent changes in the person's behavior. Therefore, this invention also analyzes brightness changes in the background area and compensates for changes in the person area, reducing the impact of background factors on the keyframe recognition results and improving the accuracy of keyframe extraction. Sentiment analysis is performed on the audio sequence to obtain a set of character emotion sequences. Since the audio in the multimedia to be identified typically contains multiple sound sources such as human voices, background music, and environmental noise, directly analyzing human emotions based on the mixed audio is easily affected by interference from other sound sources, leading to inaccurate emotion recognition. Therefore, this invention performs noise reduction and sound source separation on the audio sequence to obtain candidate speech sequences corresponding to different sound sources, and determines the target human voice sequence by combining the correlation between changes in mouth movements and speech activity time in the video. This invention extracts acoustic emotion features from the target human voice sequence and obtains the human emotional state by combining speech text information and the human's visual performance in the video. Human target region detection is performed on the keyframe sequence to obtain multiple key human region sets, and human interaction feature extraction is performed on the keyframe sequence based on multiple key human region sets to obtain human interaction feature vectors. Since violent behavior typically manifests as dynamic relationships such as approach, contact, and interaction between individuals, existing technologies, relying solely on changes in human movement, easily misidentify non-violent scenes like sports collisions, film actions, and normal interactions as violent behavior. Therefore, this invention obtains the trajectory of human targets by cross-frame correlation across multiple keyframes and extracts human interaction features based on changes in human movement, approach, and contact. This describes the behavioral state from the perspective of dynamic interaction relationships between individuals, improving the ability to distinguish between violent and non-violent behavior. Furthermore, this invention performs content recognition on the multimedia to be identified based on a set of human emotion sequences and human interaction feature vectors, obtaining the multimedia content recognition result.Because violent behavior encompasses multiple aspects, including changes in character actions, interactions, emotional shifts in speech, and semantic changes in text, relying solely on a single modality for judgment is susceptible to factors such as scene similarity, semantic gaps, and environmental interference. This can lead to misidentification and false bans during online live streaming and video monitoring. Therefore, this invention integrates character emotion sequences and character interaction feature vectors to comprehensively analyze the multimedia content to be identified from multiple dimensions, including character behavior, emotions, and semantic content, thereby improving the accuracy of multimedia content recognition in complex scenarios. Thus, this invention can adaptively determine key video frames based on changes in character behavior within the video content and integrate information from multiple modalities, including video, speech, and text, to enhance the accuracy of violent behavior recognition. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a multimedia content recognition method based on cross-modal feature fusion provided in an embodiment of the present invention. Figure 2 A functional block diagram of a multimedia content recognition system based on cross-modal feature fusion provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the multimedia content recognition method based on cross-modal feature fusion, according to an embodiment of the present invention.

[0020] Explanation of reference numerals in the attached figures: 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0021] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] This application provides a multimedia content recognition method based on cross-modal feature fusion. The executing entity of the multimedia content recognition method based on cross-modal feature fusion includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the multimedia content recognition method based on cross-modal feature fusion can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0024] Reference Figure 1The diagram shown is a flowchart illustrating a multimedia content recognition method based on cross-modal feature fusion according to an embodiment of the present invention. In this embodiment, the multimedia content recognition method based on cross-modal feature fusion includes: S1. Obtain the multimedia to be identified, extract the video frame sequence and audio sequence from the multimedia to be identified, perform key frame recognition on the video frame sequence, and obtain the key frame sequence.

[0025] It should be noted that the multimedia to be identified refers to the multimedia that needs to be identified in terms of content recognition. Content recognition refers to identifying the content of the multimedia. For example, it is necessary to identify whether the content in a short video (multimedia) is violent.

[0026] It is understood that extracting the video frame sequence and audio sequence from the multimedia to be identified refers to decapsulating the multimedia to be identified into audio and video, then performing video decoding to obtain multiple video frames, arranging the multiple video frames in chronological order according to the timestamps corresponding to each video frame to obtain a video frame sequence, and performing audio decoding to obtain an audio signal. The decapsulation, video decoding, and audio decoding are all existing technologies, and will not be described in detail here.

[0027] It needs to be explained that with the continuous increase in the amount of multimedia content such as short videos and live streams, online platforms need to monitor a large amount of user-posted multimedia content for identification in real-time or near real-time. Content such as violent behavior only exists in a portion of the video frames within the multimedia to be identified. Since the video frame sequence is extracted from a complete multimedia content, it contains a large amount of repetitive content. Directly performing person detection, motion analysis, and cross-modal feature fusion on the video frame sequence would increase the data processing volume and reduce content recognition efficiency. Therefore, it is necessary to perform keyframe recognition on the video frame sequence to filter video frames that reflect changes in human motion from a continuous sequence.

[0028] Furthermore, the step of performing keyframe recognition on the video frame sequence to obtain a keyframe sequence includes: For each video frame in the video frame sequence, human region recognition is performed to obtain multiple sets of human target regions, where each set of human target regions corresponds one-to-one with a video frame. A sliding window extraction operation is performed on the video frame sequence to obtain multiple video frame combinations, wherein the video frame combination includes two combined video frames; Extract one video frame combination from multiple video frame combinations sequentially to obtain the target video frame combination, and perform the following operations on the target video frame combination: Extract two target person region sets corresponding to the combination of target video frames from multiple target person region sets; Determine the difference region of the target person based on the two target person region sets; Extract two image regions of the target person based on the difference region of the target person; Calculate the grayscale difference between two images of a person's region to obtain the amount of change in the person's region; Keyframes are obtained by performing keyframe detection based on changes in the character's area. The keyframes are extracted and summarized in chronological order by using a sliding window that combines multiple video frames to obtain a keyframe sequence.

[0029] It should be understood that the process of performing person region recognition on each video frame in the video frame sequence refers to identifying a set of person target regions for each video frame using existing object detection models such as YOLO. The set of person target regions includes one or more person target regions, which refer to the regions where the person is located in the video frame. It should be noted that the person target regions output by existing object detection models such as YOLO are represented using rectangular bounding boxes.

[0030] Understandably, the present invention determines keyframes by the difference region of the target person between two video frames. Therefore, this embodiment performs the sliding window extraction operation on the video frame sequence to obtain multiple video frame combinations. The specific operation is as follows: using two video frames as the length of the sliding window and one video frame as the step size, multiple video frame combinations are extracted from the video frame sequence.

[0031] It is clear that the target region set of the person corresponds one-to-one with the video frame, and the two combined video frames included in the target video frame combination are two video frames extracted from the video frame sequence. Therefore, two target region sets of the person corresponding to the two combined video frames in the target video frame combination can be extracted from multiple target region sets of the person as two target region sets, and the target region sets correspond one-to-one with the combined video frames.

[0032] Specifically, the step of determining the target person difference region based on two target person region sets refers to extracting the pixel coordinate set of the target person region from the video frame corresponding to each target person region set for each target person region set, summarizing the pixel coordinate sets to obtain the pixel coordinate set corresponding to the target person region set, finding the union of the two pixel coordinate sets corresponding to the two target person region sets to obtain the pixel coordinate union, and taking the region formed by multiple pixel coordinates in the pixel coordinate union as the target person difference region.

[0033] It should be noted that, because a person's posture changes during the execution of violent acts, the target person regions represented by the same person in two adjacent video frames may differ. For example, in one video frame, a person's arm may be obscured by their body and not detected, while in an adjacent video frame, the person's arm may extend beyond the original target person region, but the original target person region remains largely unchanged. In this case, if only the target person region set corresponding to the aforementioned video frame is used to calculate the change in the person region, the changes in the newly added person region in the adjacent video frames may be missed. For example, if the change in the target person's region is calculated using the target person's region set in a video frame where the arm is obscured, and the change in the target person's region set in an adjacent video frame is only the appearance of the arm, with no change in the other target person's regions, then the calculated change in the person's region will be 0. This would lead to the assumption that the two adjacent video frames are duplicates and not keyframes, even though the target person's region set has actually changed. Therefore, this embodiment uses the above operation to find the union of two sets of pixel coordinates to determine the total area occupied by the person in the two video frames combined with the target video frames, and then calculates the change in the person's region, avoiding the omission of changes in the person in the video frames.

[0034] In detail, based on the pixel coordinates corresponding to the differential region of the target person, the same pixel coordinates are extracted from each combined video frame. The area represented by these same pixel coordinates is the total area occupied by the person. The gray value corresponding to the pixel coordinate is then assigned to the position of the same pixel coordinate in the total area occupied by the person to obtain the image of the person region.

[0035] It should be noted that, in order to eliminate the changes in grayscale values ​​caused by background factors such as lighting, this invention performs brightness compensation on the two human figure area images to obtain two compensated human figure area images.

[0036] Furthermore, the calculation of the grayscale difference between the two person region images to obtain the change in the person region includes: Determine the foreground background region and the background region. Extract multiple grayscale values ​​of foreground background pixels from the foreground background region and extract multiple grayscale values ​​of background pixels from the background region. The grayscale values ​​of foreground background pixels and background background pixels correspond one-to-one. Multiple background grayscale values ​​are obtained by subtracting multiple front background pixel grayscale values ​​from multiple back background pixel grayscale values. Extract the background brightness change from multiple background grayscale changes; Calculate the image difference between the two images of the person's region to obtain the difference image; Extract multiple difference grayscale values ​​from the difference image, and calculate the average of the absolute values ​​of the multiple difference grayscale values ​​to obtain the amount of grayscale change of the person; The change in the figure's grayscale is obtained by subtracting the change in the background brightness.

[0037] Specifically, the foreground background region refers to the background region in the foreground target video frame, and the background region refers to the background region in the background of the background target video frame. The background region refers to the area in the two combined video frames excluding the human figure image. Since the two human figure images in the two combined video frames represent the same area, the foreground background region and the background region represent the same area, and the pixel coordinates in the foreground background region are the same as those in the background region. The foreground target video frame refers to the earlier combined video frame in the video frame sequence among the two combined video frames of the target video frame combination, and the background region refers to the later combined video frame in the video frame sequence among the two combined video frames of the target video frame combination.

[0038] Specifically, the grayscale value of the foreground background pixel refers to the grayscale value at a pixel coordinate in the foreground background region, and the grayscale value of the background pixel refers to the grayscale value at a pixel coordinate in the background region. The background grayscale change refers to the difference between the grayscale value of the background pixel and the corresponding grayscale value of the foreground background pixel, representing the change in grayscale value of the background region in two adjacent video frames.

[0039] Importantly, since the movement of objects in the background can also cause differences in the grayscale values ​​of the foreground and background pixels, such as the movement of leaves or curtains, the grayscale values ​​of the foreground and background pixels at the same location will differ. Therefore, this invention sorts the absolute values ​​of multiple background grayscale changes in ascending order to obtain a grayscale change sequence. Then, according to a preset percentile (e.g., 20%), the top 20% of the background grayscale changes are extracted from the grayscale change sequence to calculate the background brightness change, thus avoiding interference from the background grayscale changes caused by the movement of objects.

[0040] It should be explained that, since the difference in grayscale values ​​caused by the movement of objects in the background includes the difference in grayscale values ​​caused by the background light source, and the difference in grayscale values ​​caused by the movement of objects in the background is greater than the difference in grayscale values ​​caused by the background light source, taking the absolute value of multiple background grayscale changes and sorting them can help select the background grayscale change amount that only represents the difference in grayscale values ​​caused by the background light source. The background brightness change amount refers to the average of multiple background grayscale change amounts, representing the change in grayscale values ​​generated in the background due to factors such as lighting. The calculation of the image difference between two person area images refers to subtracting the grayscale value at the same pixel coordinate position in the person area image corresponding to the previous target video frame from each grayscale value in the person area image corresponding to the subsequent target video frame.

[0041] It is understood that the difference grayscale value refers to the grayscale value of a single pixel in the difference image. The grayscale change of the person refers to the average of the absolute values ​​of multiple difference grayscale values, representing the amount of change in the grayscale value of the target area of ​​the person. The change in the person area represents the amount of change in the grayscale value of the target area of ​​the person after removing the background brightness change, representing the actual amount of change in the grayscale value of the target area of ​​the person.

[0042] It should be noted that the specific implementation of the key frame judgment based on the change amount of the character area to obtain the key frame is as follows: firstly, extract the target video frame combination where the change amount of the character area is greater than or equal to the preset threshold of the change amount of the character area, and record the subsequent target video frame in the target video frame combination as the key frame.

[0043] It should be understood that the specific implementation of extracting keyframes in the order of first to last according to the sliding window of multiple video frame combinations to obtain a keyframe sequence is as follows: according to the extraction order when extracting video frame combinations by the sliding window, the keyframes corresponding to the extracted video frame combinations are arranged to obtain a keyframe sequence.

[0044] S2. Perform sentiment analysis on the audio sequence to obtain a set of character emotion sequences.

[0045] Furthermore, the step of performing sentiment analysis on the audio sequence to obtain a set of character emotion sequences includes: Perform source separation on the audio sequence to obtain multiple candidate speech source sequences; Perform voice association on multiple candidate sound source speech sequences to obtain multiple target voice sequences; Speech segmentation is performed on multiple target person speech sequences to obtain multiple speech segment sets, where each speech segment set corresponds one-to-one with a target person speech sequence and includes multiple speech segments. For each of the multiple speech segment sets, perform the following operation: Identify multiple multimodal emotion feature groups based on a set of speech segments; Emotional state recognition is performed on multiple multimodal emotion feature groups to obtain a sequence of human emotional states, wherein the sequence of human emotional states includes multiple human emotional states, and each human emotional state corresponds one-to-one with a speech segment. By summarizing the sequences of characters' emotional states, we obtain a set of character emotional sequences.

[0046] Furthermore, the audio sequence is subjected to source separation to obtain multiple candidate source speech sequences, including: Denoising is performed on the audio sequence to obtain the denoised audio sequence; Perform time-frequency transformation on the noise-reduced audio sequence to obtain a frequency domain audio sequence, and extract audio time-frequency features from the frequency domain audio sequence; The audio time-frequency features are decomposed to obtain multiple sound source spectral features; Speech reconstruction is performed on the spectral features of multiple sound sources to obtain multiple candidate sound source speech sequences.

[0047] Understandably, the audio sequence is an audio signal extracted from the multimedia to be identified. Since the audio signal in the multimedia to be identified is usually formed by the superposition of multiple sound sources such as multiple voices, background music, and environmental noise, the waveforms of the audio signals from different sound sources are mixed with each other, making it difficult to directly extract the emotional state of each person from the audio sequence. Therefore, this invention first performs sound source separation on the audio sequence to obtain the candidate sound source speech sequence corresponding to each sound source.

[0048] Specifically, because audio sequences are affected by background noise during the acquisition process, this invention uses existing filtering algorithms such as Wiener filtering to denoise the audio sequences and obtain denoised audio sequences.

[0049] Importantly, the sound source separation involves separating candidate speech sequences corresponding to different sound sources from a noise-reduced frequency sequence containing a mixture of different sound sources. In general existing technologies, sound source separation can be achieved by utilizing the different frequencies of the sound sources. However, the noise-reduced frequency sequence in this embodiment is a discrete-time signal. The amplitude at each sampling moment is the result of the superposition of the amplitudes of multiple sound sources at that moment, which cannot reflect the amplitudes of different sound sources at different sampling moments and different frequencies. Therefore, this embodiment performs time-frequency conversion on the noise-reduced frequency sequence, converting the discrete-time signal into a frequency-domain signal (i.e., a frequency-domain audio sequence), and extracts the amplitude spectrum from the converted frequency-domain signal as the audio time-frequency feature. This process of obtaining the audio time-frequency feature can be implemented using existing technologies such as short-time Fourier transform, which will not be elaborated upon here.

[0050] It should be explained that the amplitude spectrum in the audio time-frequency feature represents the energy distribution of the audio signal corresponding to different times and frequencies. Since different sound sources (such as different voices, background music and environmental sounds) have different sound emission methods and frequency compositions, they have different energy distributions in the amplitude spectrum. Therefore, the sound source decomposition of the audio time-frequency feature described in this embodiment refers to: using existing amplitude spectrum decomposition algorithms such as non-negative matrix decomposition to perform matrix decomposition on the amplitude spectrum in the audio time-frequency feature, decomposing the mixed amplitude spectrum into multiple sound source spectral features corresponding to multiple independent sound sources. Each sound source spectral feature is used to describe the audio signal of a sound source.

[0051] It should be understood that the speech reconstruction performed on the spectral features of multiple sound sources refers to recovering the corresponding time-domain audio signal based on the spectral features of each sound source. This can be achieved through existing time-frequency inverse transformation algorithms such as inverse short-time Fourier transform, which will not be elaborated upon here.

[0052] Furthermore, the step of performing person-to-person speech association on multiple candidate sound source speech sequences to obtain multiple target person speech sequences includes: Extract multiple mouth region sequences from multiple target regions of individuals; Cross-frame mouth motion analysis was performed on multiple mouth region sequences to obtain multiple mouth motion feature sequences; Multiple mouth movement time segments were determined based on multiple mouth movement feature sequences; Speech activity detection is performed on speech sequences from multiple candidate sound sources to obtain multiple sets of speech activity time periods; Based on multiple mouth movement time periods and multiple speech activity time periods, person association is performed on multiple candidate sound source speech sequences to obtain multiple target person speech sequences.

[0053] Understandably, the character speech association is used to determine that the candidate sound source speech sequence belongs to the speech of each character in the multimedia to be identified, and then to identify the character's emotional state sequence based on the candidate sound source speech sequence corresponding to the character.

[0054] Specifically, the implementation process of extracting multiple mouth region sequences from multiple target regions of people is as follows: For each person, extract the target region set corresponding to each person from the multiple target regions of people set, and arrange the multiple target regions of people in the target region set (each target region corresponds to a video frame); arrange the multiple video frames corresponding to the multiple target regions of people in the video frame sequence to obtain a target region sequence, wherein each target region sequence of people includes multiple target region sequences, and one target region sequence corresponds to one person's target region in one video frame; perform facial key point detection on each target region sequence of people to determine the mouth region; summarize the mouth regions according to the order of the multiple target regions sequence of people in the target region sequence to obtain a mouth region sequence; summarize the mouth region sequences to obtain the mouth region sequences corresponding to multiple people. The facial key point detection can be implemented using existing facial key point detection algorithms such as the 68-point facial key point detection algorithm, which will not be elaborated here.

[0055] Furthermore, the step of performing cross-frame mouth motion analysis on multiple mouth region sequences yields multiple mouth motion feature sequences, including: Extract a mouth region sequence from multiple mouth region sequences sequentially to obtain the target mouth region sequence, and then perform the following operations on the target mouth region sequence: Perform lip keypoint detection on each target mouth region in the target mouth region sequence to obtain multiple lip keypoint sets, where each lip keypoint set corresponds one-to-one with a target mouth region; Based on multiple lip keypoint sets, position normalization and scale normalization are performed on each target mouth region in the target mouth region sequence to obtain a normalized mouth region sequence. A sliding window extraction operation is performed on the normalized mouth region sequence to obtain multiple combinations of adjacent mouth regions; Extract one adjacent mouth region combination from multiple adjacent mouth region combinations sequentially to obtain the target adjacent mouth region combination, and perform the following operations on the target adjacent mouth region combination: Lip key point detection is performed on the combination of adjacent mouth regions of the target to obtain two target lip key point sets. Based on the two target lip key point sets, the lip opening and closing change characteristics, the relative displacement characteristics of lip key points, and the mouth shape change characteristics are calculated. Vector concatenation is performed on the lip opening and closing characteristics, the relative displacement characteristics of key lip points, and the mouth shape change characteristics to obtain mouth motion features; Based on the extraction of mouth motion features in a sliding window combination of multiple adjacent mouth regions, the mouth motion features are summarized in order from first to last to obtain the mouth motion feature sequence. By summarizing the mouth movement feature sequences, multiple mouth movement feature sequences are obtained.

[0056] Understandably, since a single mouth region can only reflect the static state of a person's mouth in a single video frame, it cannot characterize the opening, closing, displacement, and shape changes of the lips across multiple video frames. Therefore, it is necessary to perform cross-frame mouth motion analysis on each mouth region sequence separately to extract the changes in each person's mouth over time and obtain multiple mouth motion feature sequences.

[0057] Specifically, the lip key point set includes points on the contour of the corner of the mouth, such as the upper lip key point, lower lip key point, left corner of the mouth key point, and right corner of the mouth key point. It can be implemented by existing face key point detection algorithms such as the 68-point face key point detection algorithm, which will not be elaborated here.

[0058] Importantly, the mouth area of ​​the same person can change in position and size across different video frames due to factors such as head translation and body movement. Directly comparing the original mouth areas in different video frames can easily lead to misinterpreting changes caused by head movement as lip movement. Therefore, it is necessary to transform the mouth area in different video frames to a unified position and scale reference.

[0059] In detail, the average of the two x-coordinates of the pixel coordinates of the left and right corner keypoints in each lip keypoint set is used as the x-coordinate of the mouth center position, and the average of the two y-coordinates of the pixel coordinates of the left and right corner keypoints is used as the y-coordinate of the mouth center position, thus obtaining the pixel coordinates of the mouth center position. The Euclidean distance between the left and right corner keypoints is then calculated as the mouth reference scale.

[0060] In more detail, multiple mouth regions corresponding to the first video frame in the video frame sequence are obtained from the mouth region sequence. The center position of each mouth region in each of these multiple mouth regions is used as a position reference. For the remaining video frames in the video frame sequence excluding the first video frame, the center position of the mouth region in each of the remaining video frames is moved to the position reference according to the correspondence between the mouth region and the target area of ​​the person. The target mouth region is translated so that the center positions of the mouths of multiple target mouth regions are at the same pixel coordinate, thereby transforming the mouth regions in different video frames to a unified position reference and completing position normalization.

[0061] Furthermore, multiple mouth regions corresponding to the first video frame in the video frame sequence are obtained in the mouth region sequence. Each mouth reference scale of each mouth region in these multiple mouth regions is used as a scale benchmark. For the remaining video frames in the video frame sequence other than the first video frame, the mouth reference scale of each mouth region in each remaining video frame is divided by the corresponding scale benchmark to obtain a scaling factor. Then, the scaling factor is multiplied by the x-coordinate and y-coordinate of the pixel coordinates of the corresponding lip keypoints in the lip keypoint set to complete the scale normalization and obtain the normalized mouth region sequence.

[0062] It should be understood that the sliding window extraction operation on the normalized mouth region sequence refers to extracting multiple combinations of adjacent mouth regions from the normalized mouth region sequence using a sliding window length of two normalized mouth regions and a step size of one normalized mouth region. Each combination of adjacent mouth regions includes two adjacent mouth regions, which are the normalized mouth regions extracted by the sliding window. The lip keypoint detection operation on the target adjacent mouth region combination refers to performing lip keypoint detection on each target adjacent mouth region in the target adjacent mouth region combination. This lip keypoint detection process is consistent with the process of performing lip keypoint detection on each target mouth region in the target mouth region sequence, and will not be elaborated further here.

[0063] It is important to understand that the lip opening / closing variation feature is used to describe the change in the Euclidean distance between the upper and lower lips between two adjacent video frames. Specifically, the Euclidean distance between the corresponding upper and lower lip keypoints in two target lip keypoint sets can be calculated separately, and then the absolute value of the difference between the two Euclidean distances can be calculated to obtain the lip opening / closing variation feature. The lip opening / closing variation feature indicates the degree of change in the lip opening / closing state.

[0064] It is understood that the relative displacement feature of the lip keypoints is used to describe the relative motion of keypoints such as the left corner of the mouth keypoint, the upper lip keypoint, and the lower lip keypoint between two adjacent video frames. Specifically, a target lip keypoint is extracted from one set of target lip keypoints, and then target lip keypoints at the same position are extracted from another set of target lip keypoints (for example, the left corner of the mouth keypoint is extracted from two video frames as the target lip keypoint). The Euclidean distance between the two target lip keypoints at the same position is calculated, resulting in multiple Euclidean distances corresponding to multiple target lip keypoints in the target lip keypoint set. The multiple Euclidean distances are then concatenated into vectors to obtain the relative displacement feature of the lip keypoints. The relative displacement feature of the lip keypoints includes the displacement of each lip keypoint in the set of lip keypoints between two adjacent video frames.

[0065] In detail, the mouth shape change feature is used to describe the changes in the contour of the mouth region between two adjacent video frames. Specifically, mouth shape parameters (such as mouth width, mouth aspect ratio, and lip contour area) can be calculated based on target lip keypoints, and the changes in these mouth shape parameters between two adjacent video frames can be calculated to obtain multiple changes corresponding to different mouth shape parameters. These multiple changes are then vector-concatenated to obtain the mouth shape change feature. The mouth width can be obtained by calculating the Euclidean distance between the left and right corner keypoints in a video frame. The mouth aspect ratio is used to characterize the mouth shape change. The Euclidean distance between the upper and lower lip keypoints is calculated, and this Euclidean distance is divided by the mouth width (which can be obtained by calculating the Euclidean distance between the left and right corner keypoints in the video frame) to obtain the mouth aspect ratio. The lip contour area is used to characterize the changes in the lip region. Multiple target lip key points in a video frame are connected sequentially with straight lines to form a closed contour. The area of ​​the closed contour is calculated using a polygon area calculation method, and this area is used as the lip contour area. The polygon area calculation method is existing technology and will not be described in detail here.

[0066] It is understood that performing vector concatenation on the lip opening and closing change features, the relative displacement features of key lip points, and the mouth shape change features means concatenating the lip opening and closing change features, the relative displacement features of key lip points, and the mouth shape change features into a vector to obtain the mouth movement features.

[0067] It should be understood that the mouth motion feature sequence includes multiple mouth motion features corresponding to the same target person, and the multiple mouth motion features are arranged in order from first to last according to the sliding window extraction of the corresponding adjacent mouth region combination. Each mouth motion feature is used to describe the changes in the mouth region, such as the opening and closing of the lips and the changes in the shape of the mouth, that occur between two adjacent video frames of the target person.

[0068] Furthermore, the present invention determines multiple mouth movement time period sets based on multiple mouth movement feature sequences, and associates candidate sound source speech sequences with people by the degree of correlation between mouth movement time period sets and speech activity time period sets.

[0069] Specifically, a mouth movement feature sequence is extracted sequentially from multiple mouth movement feature sequences to obtain a target mouth movement feature sequence. This target mouth movement feature sequence is then input into a pre-constructed mouth movement state recognition model. The model is used to classify the multiple mouth movement features in the target sequence, resulting in multiple mouth movement states corresponding to these features. These mouth movement states include both active and inactive mouth movements.

[0070] In detail, the mouth movement state recognition model can be a trained convolutional long short-term memory network model. The specific training process of this model is as follows: acquire multiple video samples containing human speaking and non-speaking behaviors, extract the mouth region from the video samples to obtain multiple mouth region sequences, manually label the multiple mouth region sequences to obtain corresponding mouth movement state labels, input the multiple mouth region sequences into the untrained mouth movement state recognition model to obtain the predicted mouth movement state, use the manually labeled mouth movement state labels as the true labels, and minimize the cross-entropy loss function between the true labels and the predicted mouth movement states using the Adam algorithm to complete the training of the mouth movement state recognition model.

[0071] It should be noted that, according to the order of multiple mouth movement features in the target mouth movement feature sequence, multiple mouth movement states are arranged to obtain a mouth movement state sequence. From the mouth movement state sequence, multiple mouth movement states that are currently in motion are extracted, resulting in multiple currently in motion mouth states. The moment of each currently in motion mouth state is determined, resulting in multiple motion moments. Multiple consecutive motion moments are combined into a mouth movement time period, resulting in a set of mouth movement time periods corresponding to the target mouth movement feature sequence.

[0072] It should be explained that the multiple mouth movement time period sets are used to represent the time range in which multiple target figures make mouth movements in a visual sense, so that subsequent time correlation calculations can be performed with the multiple speech activity time period sets corresponding to the multiple candidate sound source speech sequences to determine the multiple figures corresponding to the multiple candidate sound source speech sequences.

[0073] Understandably, the candidate sound source speech sequence is the audio signal corresponding to the candidate sound source obtained after performing sound source separation on the audio sequence. Since each candidate sound source speech sequence typically has the same duration as the original audio sequence and includes the speech activity interval, non-speech interval, residual noise, and residual sound source separation signal of the candidate sound source, it is impossible to directly determine the actual time of speech activity of the candidate sound source based on the complete candidate sound source speech sequence. To obtain audio time information that can be temporally correlated with the set of mouth movement time periods corresponding to the target person, it is necessary to perform speech activity detection on multiple candidate sound source speech sequences separately to determine multiple time ranges in which valid person speech actually exists within the multiple candidate sound source speech sequences.

[0074] Specifically, the step of performing speech activity detection on multiple candidate sound source speech sequences refers to using existing technologies such as speech activity detection for each candidate sound source speech sequence to determine multiple time ranges in each candidate sound source speech sequence where human speech exists as multiple speech activity time periods, and recording the multiple speech activity time periods as a speech activity time period set.

[0075] Importantly, the step of performing person association pointer on multiple candidate sound source speech sequences involves extracting the corresponding mouth movement time period set for each candidate sound source speech sequence, calculating the dynamic time warping of the mouth movement time period set with each speech activity time period set, obtaining multiple dynamic time warps, and extracting the mouth movement time period set corresponding to the maximum value among the multiple dynamic time warps as the matching time period set. The person corresponding to the matching time period set is taken as the person corresponding to the candidate sound source speech sequence, and the candidate sound source speech sequence of the corresponding person is recorded as the target person speech sequence.

[0076] It should be explained that the target person's speech sequence includes audio signals of a person at multiple sampling moments of the multimedia to be identified. Different time periods correspond to different emotional states of the person. If emotion recognition is directly performed on the target person's speech sequence, it is easy to merge the emotional states of the person at multiple time periods. This makes it impossible to align with the person's changing interaction features at multiple moments in time, which may lead to failure in identifying violent behavior. For example, in the target person's speech sequence, 0 seconds to 10 seconds correspond to normal communication, during which the person is in a calm state. 10 seconds to 15 seconds correspond to the person arguing, with the person's voice showing increased volume and faster speech rate, corresponding to an angry state. 15 seconds to 25 seconds correspond to the person continuing to explain, returning to a calm state. If emotion recognition is directly performed on the complete target person's speech sequence from 0 seconds to 25 seconds, the extracted acoustic features may simultaneously include the fundamental frequency, intensity, and speech rate changes corresponding to both the calm and angry states, making it difficult for the model to determine the specific time and location of the angry emotion. At this point, even if the video detects interaction such as a person approaching or touching another person for 10 to 15 seconds, it cannot be determined whether this time period corresponds to the person's angry emotions. This results in the inability to effectively integrate the person's emotional characteristics with their interaction characteristics, thus affecting the results of violent behavior recognition. Therefore, it is necessary to perform speech segmentation on each target person's speech sequence to obtain a set of speech segments, and then identify the person's emotional state corresponding to each speech segment in each speech segment set. Specifically, the speech segmentation of multiple target person speech sequences can be performed through the aforementioned speech activity detection, which is existing technology and will not be elaborated upon here.

[0077] Furthermore, the step of identifying multiple multimodal emotion feature groups based on the speech segment set includes: Extract speech segments one by one from the speech segment set to obtain the target speech segment, and perform the following operations on the target speech segment: Extract the video frame subsequence corresponding to the target audio segment from the video frame sequence, and obtain the audio character region sequence of the video frame subsequence; Acoustic emotion feature extraction is performed on the target speech segment to obtain the target acoustic emotion features; Speech recognition is performed on the target speech segment to obtain the target text segment. Then, text emotion features are extracted from the target text segment to obtain the target text emotion features. Visual emotion features are extracted from the speech character region sequence to obtain the target visual emotion features; The target acoustic emotion features, target text emotion features, and target visual emotion features are denoted as the multimodal emotion feature group; By summarizing the multimodal emotion feature groups, multiple multimodal emotion feature groups are obtained.

[0078] It should be noted that extracting the video frame subsequence corresponding to the target audio segment from the video frame sequence refers to extracting multiple video frames within the same time range from the video frame sequence based on the start and end times of the target audio segment. For example, if the start time of the target audio segment is the 3rd second and the end time is the 6th second, then multiple video frames between the 3rd and 6th seconds are extracted from the video frame sequence to obtain the video frame subsequence corresponding to the target audio segment.

[0079] It is understood that the speech character region sequence refers to the sequence obtained by arranging the multiple character target regions in the video frame sub-sequence corresponding to the target speech segment according to the order of the corresponding multiple video frames in the video frame sub-sequence.

[0080] In detail, the aforementioned acoustic emotion feature extraction of the target speech segment refers to extracting acoustic features that can reflect a person's emotional state from the sound performance of the target speech segment. These target acoustic emotion features may include, but are not limited to, fundamental frequency, intensity, speech rate, pause duration, Mel-frequency cepstral coefficients, and spectral variation features. For example, when a person raises their volume, speeds up their speech, or increases their pitch during an argument, the corresponding intensity, speech rate, and fundamental frequency will change; these changes can be used to reflect the person's emotions while speaking.

[0081] Specifically, performing speech recognition on the target speech segment to obtain the target segment text refers to using existing speech recognition technology to convert the voice of the person in the target speech segment into text as the target segment text.

[0082] It should be explained that the text emotion feature extraction of the target text segment refers to extracting features that reflect the character's emotions from the words and semantics of the target text segment. These target text emotion features may include, but are not limited to, word features, sentence semantic features, emotion tendency features, and violence-related semantic features. For example, phrases such as "I'm going to hit you," "Don't come any closer," and "I'm very angry" may respectively contain semantic information related to threat, fear, or anger. The text emotion feature extraction can be implemented using existing models such as the BERT model, and this invention does not limit it.

[0083] It should be noted that the step of extracting visual emotion features from the speech character region sequence refers to performing facial landmark detection (which can be achieved through 68-point facial landmark detection) on each speech character region in the speech character region sequence to obtain a set of facial landmarks. This set of facial landmarks includes multiple facial landmark groups, each corresponding one-to-one with a speech character region and a video frame. Based on the set of facial landmarks, optical flow estimation is performed on the speech character region sequence to obtain visual emotion features. These visual emotion features characterize the dynamic motion information of a person's facial expressions over time, such as the amplitude and direction of movement of facial areas like eyebrows, eyes, and mouth, reflecting emotional states such as anger, fear, and surprise. Changes in facial expressions can be achieved using existing optical flow estimation algorithms such as the Horn-Schunck optical flow algorithm, which will not be elaborated upon here.

[0084] Importantly, when performing emotion recognition solely based on the target acoustic emotion features of a target speech segment, it is easily affected by the speaker's volume, speech rate, intonation habits, and background noise, leading to misidentification of normal speech, dramatic performance, or emotional narration as abnormal emotions. Therefore, this embodiment simultaneously extracts the target acoustic emotion features, target text emotion features, and target visual emotion features corresponding to the target speech segment, constructing a multimodal emotion feature set. This set can jointly describe the emotional state of the target speech segment from three aspects: vocal expression, language content, and visual performance, improving the accuracy of subsequent emotion recognition.

[0085] It should be understood that performing emotion state recognition on multiple multimodal emotion feature groups refers to inputting each multimodal emotion feature group into a trained emotion state recognition model, using the emotion state recognition model to identify the emotional state of the person corresponding to the multimodal emotion feature group, and summarizing the person's emotional state according to the order of multiple speech segments corresponding to multiple multimodal emotion feature groups in the target person's speech sequence to obtain the person's emotional state sequence.

[0086] It is understood that the emotion state recognition model can be an untrained support vector machine. The training process of the emotion state recognition model is as follows: acquire multiple speech segments and video frame subsequences from historical data to obtain multiple historical speech segments and multiple historical video frame subsequences; perform acoustic emotion feature extraction and speech recognition on each historical speech segment, and perform text emotion feature extraction on the historical text segment obtained from speech recognition to obtain historical acoustic emotion features and historical text emotion features; extract historical speech character region sequences from historical video frame subsequences, and perform visual emotion feature extraction on historical speech character region sequences to obtain historical visual emotion features; combine historical acoustic emotion features, historical text emotion features, and historical visual emotion features according to a preset feature arrangement order to obtain historical multimodal emotion feature groups; manually label the real character emotion state tags of multiple historical speech segments, use the real character emotion state tags as real labels, use multiple historical multimodal emotion feature groups as input data for the emotion state recognition model, use the model's predicted multiple character emotion states as model output, and minimize the cross-entropy loss function between the real labels and the model output through the RMSProp optimization algorithm to complete the training of the emotion state recognition model.

[0087] S3. Perform character target region detection on the keyframe sequence to obtain multiple key character region sets.

[0088] It should be noted that the process of performing person target region detection on the keyframe sequence is the same as the process of performing person region recognition on each video frame in the video frame sequence, and will not be repeated here. The key person region set includes multiple person target regions of a person in multiple keyframes.

[0089] S4. Based on multiple key character regions, perform character interaction feature extraction on the keyframe sequence to obtain character interaction feature vectors.

[0090] Furthermore, based on multiple key character region sets, the keyframe sequence is subjected to character interaction feature extraction to obtain a character interaction feature vector, including: Perform cross-frame target association on multiple key character regions to obtain the trajectories of multiple character targets; Perform pairwise combinations on the target trajectories of multiple characters to obtain multiple character interaction pairs; Extract one character interaction pair from multiple character interaction pairs sequentially to obtain the target character interaction pair, and perform the following operations on the target character interaction pair: Extract the execution center position boundary of the target character's interaction to obtain two center position sequences and two character region boundary sequences; Calculate the characteristics of human motion changes and human approach changes based on two central position sequences; Calculate the contact change characteristics of the two figures based on the boundary sequences of their respective regions; Perform vector concatenation on the character's movement change features, character's approach change features, and character's contact change features to obtain interactive action change sub-vectors; The interaction action change sub-vectors are summarized to obtain multiple interaction action change sub-vectors. The multiple interaction action change sub-vectors are then concatenated to obtain the character interaction features.

[0091] It should be noted that since the target region of a person in a single keyframe can only represent the position and area of ​​the person at the corresponding moment in that keyframe, it cannot directly reflect the movement process of the person in the keyframe sequence or the interaction between different people. Therefore, cross-frame target association is first performed on multiple sets of target regions of people. The target regions of the same person in different keyframes are associated in chronological order according to the time corresponding to the different keyframes, resulting in multiple target trajectories of people. Then, pairwise combinations are performed on multiple target trajectories of people to determine multiple person interaction pairs. For each person interaction pair, the center position and area boundary of the person in the interaction pair as a function of time are extracted, and the motion change features and proximity change features of the person are calculated to construct a person interaction feature vector, providing a data foundation for subsequent content recognition.

[0092] It should be understood that the aforementioned cross-frame target association for multiple key person region sets refers to using existing multi-target tracking algorithms such as DeepSORT to match the identities of key person regions in multiple key person region sets based on the location and region intersection-over-union ratio of key person regions in different key frames. Key person regions belonging to the same person are then associated according to the time sequence of multiple key frames corresponding to multiple key person region sets, from earliest to latest, to obtain multiple person target trajectories.

[0093] In detail, the phrase "combining multiple character target trajectories pairwise to obtain multiple character interaction pairs" refers to arbitrarily extracting two character target trajectories from the multiple character target trajectories as a character interaction pair. For example, given multiple character target trajectories: a first character target trajectory, a second character target trajectory, and a third character target trajectory, combining these multiple target trajectories pairwise yields multiple character interaction pairs: a first character interaction pair {first character target trajectory, second character target trajectory}, a second character interaction pair {first character target trajectory, third character target trajectory}, and a third character interaction pair {second character target trajectory, third character target trajectory}.

[0094] Specifically, in existing multi-target tracking algorithms such as DeepSORT, the key target region for each person's trajectory in each keyframe is represented by a rectangle. Therefore, the center position of the person's trajectory can be directly calculated using the coordinates of the four vertices of this rectangle in the image coordinate system of the keyframe corresponding to the person's trajectory. For example, the coordinates of the four vertices of the rectangle containing the person's trajectory are: the top left vertex... Top right corner vertex bottom left corner vertex And the bottom right vertex, then the coordinates of the center position of the character's target trajectory are The character target trajectory has a center position in each keyframe. Therefore, the multiple center positions corresponding to the character target trajectory in multiple keyframes can be arranged in chronological order according to the time of the different keyframes to obtain a center position sequence. There are two character target trajectories in the target character interaction pair; therefore, two center position sequences can be calculated. It should be noted that the character region boundary sequence includes the character region boundary of each character target trajectory in multiple keyframes. The character target region is represented by a rectangle; therefore, the character region boundary can be represented by two vertices [top left vertex]. and the bottom right corner vertex ].

[0095] It needs to be explained that the calculation of the character motion change features and character proximity change features based on the two center position sequences refers to the following steps: For each character target trajectory, a sliding window is extracted from the center position sequence corresponding to the character target trajectory, with the length of the two center positions as the sliding window length and the step size of the sliding window as the center position. This results in multiple center position combinations, where each center position combination includes two center positions. The Euclidean distance between the two center positions in each center position combination is calculated, resulting in multiple Euclidean distances corresponding to the multiple center position combinations. The multiple Euclidean distances are then concatenated into vectors according to the order in which the sliding window extraction is performed on the multiple center position combinations, resulting in the character target motion change vector. The character target motion change vectors are then summarized to obtain two character target motion change vectors, which are denoted as the character motion change features. For each keyframe, the coordinates of the two center positions corresponding to the target trajectories of the two characters in the keyframe are extracted, and the Euclidean distance between the two characters in the keyframe is calculated using the coordinates of the two center positions to obtain the character approach change. The character approach change is summarized to obtain the approach change of the two characters. According to the time sequence of the keyframes corresponding to the approach change of the two characters, the approach change of the character whose time in the keyframe is earlier is recorded as the preceding approach change, and the approach change of the character whose time in the keyframe is later is recorded as the following approach change. The approach change feature of the characters is obtained by subtracting the following approach change from the preceding approach change.

[0096] Importantly, the calculation of character contact change features refers to extracting the two character region boundaries corresponding to the target trajectories of the two characters in each keyframe from the two character region boundary sequences, and calculating the intersection-union ratio (IU) using the two vertices corresponding to these two character region boundaries. The IUs are then summed to obtain two IUs corresponding to the two keyframes. Based on the temporal order of the keyframes corresponding to the two IUs, the IU of the keyframe with the earlier time is recorded as the pre-IU, and the IU of the keyframe with the later time is recorded as the post-IU. The character contact change feature is obtained by subtracting the pre-IU approach change from the post-IU approach change. Calculating the IU is an existing technique in target detection IoU, and will not be elaborated upon here.

[0097] It should be understood that the character motion change feature represents the distance that two characters corresponding to the target trajectories of two characters move from one keyframe to another. The character proximity change feature represents the distance that the center position between two characters moves from one keyframe to another. The character contact change feature represents the change in the degree of overlap of the boundaries of two character regions from one keyframe to another, characterizing the possibility of contact between the two characters. The vector concatenation of the character motion change feature, character proximity change feature, and character contact change feature refers to concatenating the character motion change feature, character proximity change feature, and character contact change feature vectors into a single vector in the order of character motion change feature, character proximity change feature, and character contact change feature, thus obtaining the interactive action change sub-vector.

[0098] It is clear that the above-mentioned vector concatenation of multiple interactive action change sub-vectors refers to concatenating multiple interactive action change sub-vectors into a single vector to obtain the character's interactive features.

[0099] S5. Based on the character emotion sequence set and character interaction feature vector, perform content recognition on the multimedia to be identified to obtain the multimedia content recognition result.

[0100] It is understood that the content recognition of the multimedia to be identified refers to inputting the set of character emotion sequences and character interaction feature vectors into a pre-trained multimedia content recognition model to identify whether the multimedia to be identified contains violent behavior. The multimedia content recognition result includes two results: the presence of violent behavior and the absence of violent behavior.

[0101] It should be explained that the pre-trained multimedia content recognition model can be a long short-term memory network model. The training process of this model is as follows: for the audio sequence and video frame sequence in the multimedia to be recognized, extract the character emotion sequence set and character interaction feature vector respectively, and manually annotate the multimedia content recognition result of the multimedia to be recognized. Input the character emotion sequence set and character interaction feature vector into the untrained multimedia content recognition model, use the model to output the predicted multimedia content recognition result, use the manually annotated multimedia content recognition result as the real label, use the predicted multimedia content recognition result output by the model as the prediction result, and minimize the average error between the real label and the prediction result through the gradient descent method to complete the training of the multimedia content recognition model.

[0102] To address the problems described in the background art, this invention acquires multimedia to be identified, extracts video frame sequences and audio sequences from the multimedia, and performs keyframe recognition on the video frame sequences to obtain a keyframe sequence. Since violent behavior typically occurs only during a portion of the multimedia content, and existing technologies usually select video frames from the video frame sequence at fixed time intervals, key video frames reflecting the change in violent behavior, such as the start of an attack, the approach of a person, and contact with a person, are easily missed. Therefore, this invention performs change analysis on the target area of ​​the person in the video frame sequence, and determines keyframes based on the changes in the person area, enabling subsequent recognition processes to analyze video content that reflects changes in the person's actions. Simultaneously, since video acquisition is easily affected by factors such as changes in lighting, camera shake, and background object movement, pixel changes between adjacent video frames cannot accurately represent changes in the person's behavior. Therefore, this invention also analyzes brightness changes in the background area and compensates for changes in the person area, reducing the impact of background factors on the keyframe recognition results and improving the accuracy of keyframe extraction. Sentiment analysis is performed on the audio sequence to obtain a set of character emotion sequences. Since the audio in the multimedia to be identified typically contains multiple sound sources such as human voices, background music, and environmental noise, directly analyzing human emotions based on the mixed audio is easily affected by interference from other sound sources, leading to inaccurate emotion recognition. Therefore, this invention performs noise reduction and sound source separation on the audio sequence to obtain candidate speech sequences corresponding to different sound sources, and determines the target human voice sequence by combining the correlation between changes in mouth movements and speech activity time in the video. This invention extracts acoustic emotion features from the target human voice sequence and obtains the human emotional state by combining speech text information and the human's visual performance in the video. Human target region detection is performed on the keyframe sequence to obtain multiple key human region sets, and human interaction feature extraction is performed on the keyframe sequence based on multiple key human region sets to obtain human interaction feature vectors. Since violent behavior typically manifests as dynamic relationships such as approach, contact, and interaction between individuals, existing technologies, relying solely on changes in human movement, easily misidentify non-violent scenes like sports collisions, film actions, and normal interactions as violent behavior. Therefore, this invention obtains the trajectory of human targets by cross-frame correlation across multiple keyframes and extracts human interaction features based on changes in human movement, approach, and contact. This describes the behavioral state from the perspective of dynamic interaction relationships between individuals, improving the ability to distinguish between violent and non-violent behavior. Furthermore, this invention performs content recognition on the multimedia to be identified based on a set of human emotion sequences and human interaction feature vectors, obtaining the multimedia content recognition result.Because violent behavior encompasses multiple aspects, including changes in character actions, interactions, emotional shifts in speech, and semantic changes in text, relying solely on a single modality for judgment is susceptible to factors such as scene similarity, semantic gaps, and environmental interference. This can lead to misidentification and false bans during online live streaming and video monitoring. Therefore, this invention integrates character emotion sequences and character interaction feature vectors to comprehensively analyze the multimedia content to be identified from multiple dimensions, including character behavior, emotions, and semantic content, thereby improving the accuracy of multimedia content recognition in complex scenarios. Thus, this invention can adaptively determine key video frames based on changes in character behavior within the video content and integrate information from multiple modalities, including video, speech, and text, to enhance the accuracy of violent behavior recognition.

[0103] like Figure 2 The diagram shown is a functional block diagram of a multimedia content recognition system based on cross-modal feature fusion provided in an embodiment of the present invention.

[0104] The multimedia content recognition system 100 based on cross-modal feature fusion described in this invention can be installed in an electronic device. Depending on the functions implemented, the multimedia content recognition system 100 based on cross-modal feature fusion may include a content acquisition module 101, a keyframe recognition module 102, a sentiment analysis module 103, and a content recognition module 104. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0105] The content acquisition module 101 is used to acquire the multimedia to be identified and extract video frame sequences and audio sequences from the multimedia to be identified. The keyframe recognition module 102 is used to perform keyframe recognition on the video frame sequence to obtain a keyframe sequence. The sentiment analysis module 103 is used to perform sentiment analysis on the audio sequence to obtain a set of character emotion sequences; The content recognition module 104 is used to perform character target region detection on the keyframe sequence to obtain multiple key character region sets, extract character interaction features on the keyframe sequence based on the multiple key character region sets to obtain character interaction feature vectors, and perform content recognition on the multimedia to be recognized according to the character emotion sequence set and the character interaction feature vectors to obtain multimedia content recognition results.

[0106] In detail, the modules in the multimedia content recognition system 100 based on cross-modal feature fusion described in this embodiment of the invention employ the same methods as described above. Figure 1 The method uses the same techniques as the multimedia content recognition method based on cross-modal feature fusion described above, and can produce the same technical effects, so it will not be repeated here.

[0107] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a multimedia content recognition method based on cross-modal feature fusion, according to an embodiment of the present invention.

[0108] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a multimedia content recognition method program based on cross-modal feature fusion.

[0109] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a multimedia content recognition method program based on cross-modal feature fusion, but also to temporarily store data that has been output or will be output.

[0110] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a multimedia content recognition method program based on cross-modal feature fusion) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0111] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0112] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0113] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management system, thereby enabling functions such as charging management, discharging management, and power consumption management through the power management system. The power supply may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0114] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0115] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0116] The multimedia content recognition method program based on cross-modal feature fusion stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following: Acquire the multimedia to be identified, and extract the video frame sequence and audio sequence from the multimedia to be identified; Perform keyframe recognition on the video frame sequence to obtain the keyframe sequence; Perform sentiment analysis on the audio sequences to obtain a set of character emotion sequences; Perform character target region detection on the keyframe sequence to obtain multiple key character region sets; Based on multiple key character regions, character interaction features are extracted from keyframe sequences to obtain character interaction feature vectors. Based on the character's emotional sequence set and the character's interaction feature vector, the multimedia to be identified is subjected to content recognition, and the multimedia content recognition result is obtained.

[0117] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0118] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0119] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: Acquire the multimedia to be identified, and extract the video frame sequence and audio sequence from the multimedia to be identified; Perform keyframe recognition on the video frame sequence to obtain the keyframe sequence; Perform sentiment analysis on the audio sequences to obtain a set of character emotion sequences; Perform character target region detection on the keyframe sequence to obtain multiple key character region sets; Based on multiple key character regions, character interaction features are extracted from keyframe sequences to obtain character interaction feature vectors. Based on the character's emotional sequence set and the character's interaction feature vector, the multimedia to be identified is subjected to content recognition, and the multimedia content recognition result is obtained.

[0120] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0121] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0123] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimedia content recognition method based on cross-modal feature fusion, characterized in that, The method includes: Acquire the multimedia to be identified, and extract the video frame sequence and audio sequence from the multimedia to be identified; Perform keyframe recognition on the video frame sequence to obtain the keyframe sequence; Perform sentiment analysis on the audio sequences to obtain a set of character emotion sequences; Perform character target region detection on the keyframe sequence to obtain multiple key character region sets; Based on multiple key character regions, character interaction features are extracted from keyframe sequences to obtain character interaction feature vectors. Based on the character's emotional sequence set and the character's interaction feature vector, the multimedia to be identified is subjected to content recognition, and the multimedia content recognition result is obtained.

2. The multimedia content recognition method based on cross-modal feature fusion as described in claim 1, characterized in that, The step of performing keyframe recognition on the video frame sequence to obtain the keyframe sequence includes: For each video frame in the video frame sequence, human region recognition is performed to obtain multiple sets of human target regions, where each set of human target regions corresponds one-to-one with a video frame. A sliding window extraction operation is performed on the video frame sequence to obtain multiple video frame combinations, wherein the video frame combination includes two combined video frames; Extract one video frame combination from multiple video frame combinations sequentially to obtain the target video frame combination, and perform the following operations on the target video frame combination: Extract two target person region sets corresponding to the combination of target video frames from multiple target person region sets; Determine the difference region of the target person based on the two target person region sets; Extract two image regions of the target person based on the difference region of the target person; Calculate the grayscale difference between two images of a person's region to obtain the amount of change in the person's region; Keyframes are obtained by performing keyframe detection based on changes in the character's area. The keyframes are extracted and summarized in chronological order by using a sliding window that combines multiple video frames to obtain a keyframe sequence.

3. The multimedia content recognition method based on cross-modal feature fusion as described in claim 2, characterized in that, The calculation of the grayscale difference between two image regions of a person, to obtain the change in the person region, includes: Determine the foreground background region and the background region. Extract multiple grayscale values ​​of foreground background pixels from the foreground background region and extract multiple grayscale values ​​of background pixels from the background region. The grayscale values ​​of foreground background pixels and background background pixels correspond one-to-one. Multiple background grayscale values ​​are obtained by subtracting multiple front background pixel grayscale values ​​from multiple back background pixel grayscale values. Extract the background brightness change from multiple background grayscale changes; Calculate the image difference between the two images of the person's region to obtain the difference image; Extract multiple difference grayscale values ​​from the difference image, and calculate the average of the absolute values ​​of the multiple difference grayscale values ​​to obtain the amount of grayscale change of the person; The change in the figure's grayscale is obtained by subtracting the change in the background brightness.

4. The multimedia content recognition method based on cross-modal feature fusion as described in claim 3, characterized in that, The process of performing sentiment analysis on the audio sequence to obtain a set of character emotion sequences includes: Perform source separation on the audio sequence to obtain multiple candidate speech source sequences; Perform voice association on multiple candidate sound source speech sequences to obtain multiple target voice sequences; Speech segmentation is performed on multiple target person speech sequences to obtain multiple speech segment sets, where each speech segment set corresponds one-to-one with a target person speech sequence and includes multiple speech segments. For each of the multiple speech segment sets, perform the following operation: Identify multiple multimodal emotion feature groups based on a set of speech segments; Emotional state recognition is performed on multiple multimodal emotion feature groups to obtain a sequence of human emotional states, wherein the sequence of human emotional states includes multiple human emotional states, and each human emotional state corresponds one-to-one with a speech segment. By summarizing the sequences of characters' emotional states, we obtain a set of character emotional sequences.

5. The multimedia content recognition method based on cross-modal feature fusion as described in claim 4, characterized in that, The process of performing sound source separation on the audio sequence yields multiple candidate sound source speech sequences, including: Denoising is performed on the audio sequence to obtain the denoised audio sequence; Perform time-frequency transformation on the noise-reduced audio sequence to obtain a frequency domain audio sequence, and extract audio time-frequency features from the frequency domain audio sequence; The audio time-frequency features are decomposed into sound sources to obtain multiple sound source spectral features; Speech reconstruction is performed on the spectral features of multiple sound sources to obtain multiple candidate sound source speech sequences.

6. The multimedia content recognition method based on cross-modal feature fusion as described in claim 5, characterized in that, The process of performing voice association on multiple candidate sound source speech sequences to obtain multiple target voice sequences includes: Extract multiple mouth region sequences from multiple target regions of individuals; Cross-frame mouth motion analysis was performed on multiple mouth region sequences to obtain multiple mouth motion feature sequences; Multiple mouth movement time segments were determined based on multiple mouth movement feature sequences; Speech activity detection is performed on speech sequences from multiple candidate sound sources to obtain multiple sets of speech activity time periods; Based on multiple mouth movement time periods and multiple speech activity time periods, person association is performed on multiple candidate sound source speech sequences to obtain multiple target person speech sequences.

7. The multimedia content recognition method based on cross-modal feature fusion as described in claim 6, characterized in that, The step of performing cross-frame mouth motion analysis on multiple mouth region sequences yields multiple mouth motion feature sequences, including: Extract a mouth region sequence from multiple mouth region sequences sequentially to obtain the target mouth region sequence, and then perform the following operations on the target mouth region sequence: Perform lip keypoint detection on each target mouth region in the target mouth region sequence to obtain multiple lip keypoint sets, where each lip keypoint set corresponds one-to-one with a target mouth region; Based on multiple lip keypoint sets, position normalization and scale normalization are performed on each target mouth region in the target mouth region sequence to obtain a normalized mouth region sequence. A sliding window extraction operation is performed on the normalized mouth region sequence to obtain multiple combinations of adjacent mouth regions; Extract one adjacent mouth region combination from multiple adjacent mouth region combinations sequentially to obtain the target adjacent mouth region combination, and perform the following operations on the target adjacent mouth region combination: Lip key point detection is performed on the combination of adjacent mouth regions of the target to obtain two target lip key point sets. Based on the two target lip key point sets, the lip opening and closing change characteristics, the relative displacement characteristics of lip key points, and the mouth shape change characteristics are calculated. Vector concatenation is performed on the lip opening and closing characteristics, the relative displacement characteristics of key lip points, and the mouth shape change characteristics to obtain mouth motion features; Based on the extraction of mouth motion features in a sliding window combination of multiple adjacent mouth regions, the mouth motion features are summarized in order from first to last to obtain a mouth motion feature sequence. By summarizing the mouth movement feature sequences, multiple mouth movement feature sequences are obtained.

8. The multimedia content recognition method based on cross-modal feature fusion as described in claim 7, characterized in that, The process of identifying multiple multimodal emotion feature groups based on a set of speech segments includes: Extract speech segments one by one from the speech segment set to obtain the target speech segment, and perform the following operations on the target speech segment: Extract the video frame subsequence corresponding to the target audio segment from the video frame sequence, and obtain the audio character region sequence of the video frame subsequence; Acoustic emotion feature extraction is performed on the target speech segment to obtain the target acoustic emotion features; Speech recognition is performed on the target speech segment to obtain the target text segment. Then, text emotion features are extracted from the target text segment to obtain the target text emotion features. Visual emotion features are extracted from the speech character region sequence to obtain the target visual emotion features; The target acoustic emotion features, target text emotion features, and target visual emotion features are denoted as the multimodal emotion feature group; By summarizing the multimodal emotion feature groups, multiple multimodal emotion feature groups are obtained.

9. The multimedia content recognition method based on cross-modal feature fusion as described in claim 8, characterized in that, The process involves extracting character interaction features from keyframe sequences based on multiple key character region sets to obtain character interaction feature vectors, including: Perform cross-frame target association on multiple key character regions to obtain the trajectories of multiple character targets; Perform pairwise combinations on the target trajectories of multiple characters to obtain multiple character interaction pairs; Extract one character interaction pair from multiple character interaction pairs sequentially to obtain the target character interaction pair, and perform the following operations on the target character interaction pair: Extract the execution center position boundary of the target character's interaction to obtain two center position sequences and two character region boundary sequences; Calculate the characteristics of human motion changes and human approach changes based on two central position sequences; Calculate the contact change characteristics of the two figures based on the boundary sequences of their respective regions; Perform vector concatenation on the character's movement change features, character's approach change features, and character's contact change features to obtain interactive action change sub-vectors; The interaction action change sub-vectors are summarized to obtain multiple interaction action change sub-vectors. The multiple interaction action change sub-vectors are then concatenated to obtain the character interaction features.

10. A multimedia content recognition system based on cross-modal feature fusion, characterized in that, The system includes: The content acquisition module is used to acquire the multimedia to be identified and extract video frame sequences and audio sequences from the multimedia to be identified; The keyframe recognition module is used to perform keyframe recognition on the video frame sequence to obtain the keyframe sequence. The sentiment analysis module is used to perform sentiment analysis on audio sequences to obtain a set of character emotion sequences; The content recognition module is used to perform character target region detection on keyframe sequences to obtain multiple key character region sets. Based on the multiple key character region sets, it performs character interaction feature extraction on keyframe sequences to obtain character interaction feature vectors. Based on the character emotion sequence set and character interaction feature vectors, it performs content recognition on the multimedia to be recognized to obtain multimedia content recognition results.

Images