Short video network public opinion information identification method based on image processing technology

Abstract

The invention discloses a short video network public opinion information identification method based on an image processing technology, and relates to the technical field of artificial intelligence and image processing, and the method comprises the following steps: S001, through obtaining image frames, audio tracks and time sequence information of a short video, constructing a multi-modal fusion model, extracting continuous image frames with suspicious identity features, and carrying out the recognition of the short video network public opinion information; generating a forgery risk area distribution map; and S002, performing semantic consistency verification according to the counterfeit risk region distribution map, and extracting space and time anomaly features existing among facial micro-expressions, pronunciation actions and background semantics in the image frame. According to the method, a multi-modal model is constructed by fusing image, audio and time information, fine abnormal features, traceability forgery starting points and propagation paths in a deep forgery video are identified, and an identification strategy and a public opinion response mechanism are dynamically adjusted, so that accurate identification, adaptive processing and closed-loop control of short video public opinion risks are realized; and the identification accuracy and the treatment efficiency are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and image processing technology, specifically to a method for identifying online public opinion information in short videos based on image processing technology. Background Technology

[0002] "Short Video Online Public Opinion Information Identification Based on Image Processing Technology" refers to the use of computer vision and image processing algorithms to automatically analyze the content of image frames in short videos, thereby identifying, extracting, and judging information that may be related to online public opinion. For example, by analyzing visual elements such as people, text, scenes, iconic images, and emotional expressions in the video, combined with deep learning models and public opinion knowledge bases, it can determine whether the video involves sensitive events, negative comments, hot topics, or group emotional fluctuations. This method can achieve early detection, classification, and automatic warning of potential online public opinion risks in short video content. It is particularly suitable for addressing the challenges of rapid information dissemination, complex content, and multimodal crossover on current short video platforms, improving the efficiency and intelligence of online content supervision and public opinion governance.

[0003] The existing technology has the following shortcomings: Current image processing-based short video public opinion recognition systems still suffer from insufficient ability to identify video content synthesized using AI deepfake techniques. Specifically, after attackers use high-precision image forgery algorithms to replace faces, synthesize voices, and simulate actions of political figures, public representatives, or specific sensitive individuals, the system fails to effectively detect the falsity of these synthesized images, incorrectly identifying them as real scenes, leading to misjudgments or missed detections. This problem is particularly pronounced when the forged content is of high quality, the synthesis boundaries are finely processed, and the inter-frame consistency is adequately disguised. Furthermore, attackers often combine video compression coding mechanisms, embedding highly realistic forged images before compression and leveraging the blurring effect of image details during compression to effectively conceal typical Deepfake traces, such as edge artifacts, uneven skin tone, or abnormal blinking frequency. This makes it difficult for traditional public opinion recognition methods based on low-order feature detection to detect forged evidence.

[0004] The aforementioned deficiencies in identification capabilities could be maliciously exploited. Attackers could release fake video content to simulate inappropriate remarks or behaviors of target individuals during specific social events, sensitive public opinion moments, or international affairs. This could easily lead to public misunderstanding, emotional confrontation, and loss of control over public opinion, thereby causing a crisis of political trust, disruption of social order, and even serious consequences such as damage to the image of national governance or misjudgment of diplomatic relations.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a method for identifying online public opinion information in short videos based on image processing technology, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for identifying online public opinion information in short videos based on image processing technology, comprising the following steps: S001, by acquiring image frames, audio trajectories and time sequence information of short videos, a multimodal fusion model is constructed to extract continuous image frames with suspicious identity features and generate a distribution map of counterfeit risk areas; S002, based on the distribution map of forgery risk areas, perform semantic consistency verification and extract spatial and temporal abnormal features between facial micro-expressions, vocalizations and background semantics in the image frame; S003, based on spatial and temporal anomaly features, performs cross-frame evolution analysis to construct the temporal evolution path of the target person's facial expressions, voice, and motion trajectories; S004, based on the time evolution path, trace the source of forgery features and identify the starting image frame position and propagation path features of the forgery behavior; S005, based on the position of the starting image frame and the characteristics of the propagation path, performs recognition strategy adjustment, dynamically updates image feature extraction parameters and recognition threshold, and realizes adaptive recognition of counterfeit types; S006. The identification results after the identification strategy adjustment are input into the public opinion analysis process. The monitoring and response strategy is adjusted according to the identification confidence and evolution rate to form a closed-loop mechanism of identification, analysis and response.

[0008] Preferably, step S001 includes: The short video is decoded frame by frame at a fixed frame rate, the image frame sequence is extracted and saved in RGB three-channel format, and the original resolution is preserved. The speech signal was extracted using an audio stream parsing method, and an audio time vector was constructed based on a 16kHz sampling rate. Mel frequency cepstral coefficients, pitch fundamental frequency, short-time energy spectrum, syllable duration, speech rhythm curve, sound intensity variation value and formant distribution characteristics were extracted. A time sequence mapping relationship is established based on the timestamps of image frames and audio segments to form a time synchronization relationship between image frames and audio segments; A multimodal fusion model is constructed based on the visual features, audio trajectories, and temporal sequence information of image frames. The joint features of the multimodal models are analyzed, continuous image frame segments with suspicious identity features are extracted, and a distribution map of forgery risk areas is generated based on semantic anomalies.

[0009] Preferably, step S002 includes: High-risk image frames were read based on the distribution map of forged risk areas, and the facial region of the person was located. A facial landmark detection method based on residual neural network structure optimization is adopted to extract facial feature points and analyze facial micro-expression states. The audio signal of the corresponding time period is converted into a Mel spectrogram, which is then input into a bidirectional long short-term memory network to extract the audio features of the vocalization action, and the consistency is analyzed by aligning it with the mouth opening and closing action. Extract semantic information from the image background, determine whether there is a semantic conflict between the character's behavior and the scene composition, and integrate the three types of verification results to form the spatial and temporal anomaly risk level of the image frame.

[0010] Preferably, step S003 includes: Based on spatial and temporal anomalies, the target person's image frame sequence is extracted, and an optical flow tracking combined with a facial key point matching algorithm is used to achieve continuous tracking of the facial region. Extract facial expression features, speech frequency features, and action posture features from each frame and annotate them with a unified timestamp; Based on the sliding time window, the continuity index of behavioral characteristics on the time axis is calculated to identify abnormal mutation nodes and cross-channel collaborative anomalies. Construct a behavioral temporal evolution path in the time dimension, and mark the location of mutation nodes, feature types and difference parameters to identify the abnormal evolution process of forgery behavior.

[0011] Preferably, step S004 includes: Key behavioral mutation image frames are extracted based on the time evolution path, and a behavioral mutation index table containing timestamps, anomaly type labels, and behavioral intensity scores is generated. Starting from the earliest abnormal image frame, trace back the target person's behavior frame by frame along the timeline, and identify the last frame that meets the continuous stability condition as the starting image frame for generating the forgery behavior. Starting from the generated initial image frame, the forgery features are traced forward along the timeline to construct a forgery propagation path trajectory covering the entire video segment; The forgery source tracing analysis results are generated based on the forgery propagation path trajectory, including the starting image frame timestamp, propagation path range, forgery behavior intensity change curve, and propagation path stability score.

[0012] Preferably, step S005 includes: Extract the location of the forged starting image frame and the trajectory of its propagation path, and analyze the types, time ranges, intensity, and distribution channels of abnormal behavior. Based on abnormal behavior types and feature profiles, adjust image feature extraction parameters and audio modeling strategies to improve the feature recognition accuracy of face swapping, speech synthesis and action embedding forgery. Based on the changing trends of the duration and intensity of forgery behavior, the image recognition threshold and audio recognition threshold are dynamically updated to achieve adaptive recognition of different forgery intensity types; Configure the identification process call order according to the type of forgery behavior, construct the identification process call chain, and realize accurate adaptation and real-time identification of different types of forged content.

[0013] Preferably, step S006 includes: The confidence level, the intensity of abnormal features, and the rate of time evolution are used as structured indicators to input into the public opinion information analysis process; The risk level of counterfeit content is determined based on the confidence level of identification and the intensity of abnormal features, and the dissemination impact level assessment result is constructed by combining dissemination behavior data; Based on the level of dissemination impact, implement content review frequency adjustment, response behavior strategy activation, and account behavior archiving mechanisms; The execution results and user feedback input recognition process are combined with weight parameter optimization and strategy invocation priority updates to form a closed loop of recognition, analysis and response.

[0014] Preferably, the steps for constructing the propagation impact level assessment results include: Obtain data on the spread of the target short video across multiple platforms, including daily play counts, number of comments, number of reposts, number of likes, and the historical behavior records of the posting account; By jointly modeling the propagation behavior data with the identification confidence level, the strength of abnormal features and the rate of time evolution, we can identify whether there is cross-platform propagation, high-frequency interactive behavior or involvement of sensitive events. The risk score for the spread of information is calculated based on the breadth, speed, and intensity of interaction, and is divided into three levels of impact: highly sensitive, moderately sensitive, and low sensitive.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention constructs a multimodal joint analysis model by fusing image frames, audio trajectories, and temporal sequence information, ensuring comprehensive extraction of forgery clues even in the highly complex and structurally varied environment of short videos. Furthermore, through semantic consistency verification and cross-frame evolutionary modeling, it identifies subtle anomalies in facial expressions, voice frequencies, and gestures, enhancing the sensitivity to detect concealed forgery. Based on this, it uses the forgery origin frame and propagation path to trace the source, improving the interpretability and accuracy of forged content tracing. Finally, through dynamic identification strategy adjustment and closed-loop control of the public opinion response mechanism, it achieves intelligent and adaptive processing throughout the entire process from identification to handling. This method not only improves the accuracy and robustness of identification but also enhances the real-time nature and controllability of public opinion governance, providing technical support for platform-level video review, online content supervision, and emergency response, demonstrating significant engineering application value and social governance benefits. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0017] Figure 1 This is a flowchart of the method for identifying online public opinion information in short videos based on image processing technology according to the present invention. Detailed Implementation

[0018] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0019] This invention provides, for example Figure 1 The method for identifying online public opinion information based on short videos using image processing technology, as shown, includes the following steps: S001, acquire the image frame sequence, audio trajectory data and time sequence information of the short video, construct a multimodal fusion model based on the visual features of the image frames, audio trajectory and time sequence information, analyze the multimodal joint features, extract continuous image frame segments with suspicious identity features, and generate a distribution map of counterfeit risk areas; To achieve high-precision identification and processing of potentially fake people or fake behavior in short videos, the specific steps are as follows: The complete data structure of the target short video is obtained, including image frame sequences, audio trajectory data, and inter-frame temporal sequence information. Image frame sequences are extracted by decoding the short video frame by frame at a fixed frame rate (e.g., 25 frames / second). The extracted image frames are saved in RGB three-channel format, maintaining the original video resolution without compression to preserve edge texture and skin tone details. Audio trajectory data is extracted using audio stream parsing to obtain synchronized speech signals. An audio temporal vector is constructed based on a 16kHz sampling rate, and acoustic feature parameters such as Mel-frequency cepstral coefficients (MFCC), fundamental pitch, short-time energy spectrum, and syllable duration are extracted for each speech segment. Inter-frame temporal sequence information is obtained by analyzing the original timestamps of image frames and audio segments to establish a mapping relationship between image frame numbers and corresponding audio segments, ensuring that the content of each image frame can be associated with the speech behavior within its corresponding time period.

[0020] Based on the extracted image frames and audio trajectory data, feature extraction and temporal encoding were performed respectively. For image frames, a convolutional neural network optimized with the ResNet50 architecture was used for feature extraction. The network's forward propagation was used to calculate the distribution of key points on the face, the geometric proportions of facial features, skin tone uniformity, grayscale gradient changes in skin texture, and temporal vectors of eye opening and closing states in each frame. In addition to facial features, background stability, image edge sharpness, and the rate of change of face position were also extracted. For audio trajectories, a bidirectional long short-term memory (Bi-LSTM) network was used to model the speech sequence, obtaining syllable boundaries, peak intensity distribution, speech rate rhythm changes, and pronunciation pattern vectors. The image frame features and corresponding audio features were uniformly encoded using temporal alignment to form a timestamp-driven multimodal feature sequence, providing joint feature support for subsequent analysis.

[0021] The primary goal of modeling speech sequences using a Bidirectional Long Short-Term Memory (Bi-LSTM) network is to extract more complete contextual semantic information and temporal dependencies from the speech signal, thereby improving the accuracy and discriminative power of speech behavior representation. Traditional unidirectional LSTM networks can only learn features based on historical data up to the current time step. When processing speech sequences, certain speech features (such as stress, intonation inflections, and semantic breaks) often rely on contextual cues from future frames for accurate understanding. Bi-LSTM introduces two directional LSTM units into its network structure: one processes the sequence forward and the other backward, ensuring that the speech feature representation at each time point incorporates both past and future speech information. This approach is more sensitive to modeling complex audio behaviors such as pronunciation variations, speech rate and rhythm, pause boundaries, and emotional intonation, making it particularly suitable for identifying whether speech has been forged or edited in short video scenarios. Therefore, modeling speech sequences using Bi-LSTM not only enhances the consistency between audio features and visual frame synchronization verification but also provides more stable, context-aware acoustic feature support for subsequent forgery detection.

[0022] After initial feature encoding, a multimodal fusion mechanism is constructed to cross-model visual and audio information. Specifically, a Transformer-based cross-attention network architecture is used. The high-dimensional visual feature tensor of the input image frame sequence and the embedded representation of the speech trajectory are input. Frame order information is introduced through a positional encoding module, and semantic relevance analysis is performed through a multi-head attention mechanism to learn the temporal dependency between image semantics and speech content. The fusion result is used to construct a multimodal semantic representation matrix. Subsequently, an identity consistency judgment method is used to comprehensively determine whether there is a semantic conflict between a person's facial expression state (e.g., smiling, tension, anger), vocalization actions, and the image background within the current time period. Examples include a person speaking but remaining silent, delayed mouth movements, and inconsistencies between background lighting changes and the frequency of facial expressions. Once these behavioral characteristics are found to be inconsistent with natural temporal evolution, consecutive image frames within that time period are marked as abnormal frames and assigned an abnormality probability score.

[0023] The Transformer-based cross-attention network architecture is a deep learning structure for processing the relationships between multimodal data (such as images and speech). Its core function is to establish a cross-modal semantic alignment mechanism and temporal dependency modeling capability between visual and speech information. In this invention, the architecture first feeds high-dimensional visual feature tensors extracted from image frame sequences (such as facial keypoints, expression tensors, texture gradients, etc.) and embedded representations of speech trajectories (such as Mel-frequency cepstral coefficients, speech rhythm, etc.) into two input channels of the Transformer structure, respectively. To ensure that the network can correctly understand the sequential relationship between image frames and speech in the time series, a positional encoding module is used to add explicit temporal position markers to each frame of image features and each segment of speech features, enabling the model to perceive the temporal logic between input features. Next, a cross-attention channel is established between the image and speech feature sequences through a multi-head attention mechanism. Each attention head can learn the degree of matching between them from different semantic dimensions (such as human actions-intonation fluctuations, lip movements-syllable structure), thereby achieving semantic consistency modeling of visual and audio content at the same temporal position. Ultimately, the cross-attention network outputs a joint representation that integrates image semantics, speech features, and temporal dependencies to determine whether forgery occurs during that time period (such as lip misalignment, delayed pronunciation, or contextual disconnection), thus providing a stronger multimodal semantic foundation for subsequent forgery detection steps.

[0024] For consecutive image frames marked as anomalous, a distribution map of forgery risk areas is generated by combining their corresponding timestamps and spatial coordinates. The distribution map is represented in two dimensions, with the horizontal axis representing the image frame number and the vertical axis representing the two-dimensional pixel coordinates of the facial region within the image frame. For each image frame, the locations of detected anomalous features in key facial regions are marked, and a heatmap based on color gradients is constructed to represent the anomalous risk level. The heatmap value is calculated based on the probability score of anomalous features in the image frame and a density-weighted function of similar anomalous behaviors across multiple frame time periods. The map is updated in real-time using a sliding time window method (e.g., a 5-frame sliding window) to show the changing trends of risky behavior of the target individual over different time periods. The distribution map is stored as a structured matrix for subsequent semantic consistency verification and anomalous behavior tracking analysis.

[0025] This step serves to build an initial recognition foundation with spatiotemporal consistency and multimodal semantic linkage capabilities for subsequent fake video identification processes. By comprehensively acquiring and fusing information from image, audio, and temporal dimensions, it accurately locates high-risk areas in short videos that may contain abnormal behavior or fake content. In the dissemination of short videos, deepfake videos often embed forgery into real contexts through sophisticated face synthesis, voice cloning, and temporal camouflage. Traditional methods relying solely on single visual features or static image analysis are insufficient for effective identification. This step first acquires the image frame sequence and audio trajectory data of the video, and combines the temporal sequence information between frames to provide a precise temporal reference for multimodal feature alignment. Subsequently, visual features (such as facial key points, skin texture, and facial contours) from the image frames and acoustic features (such as speech frequency, rhythmic structure, and syllable length) from the audio are extracted separately. Deep learning methods are used to jointly model both, and semantic fusion analysis is performed on the same timeline. Finally, continuous image frames with obvious identity anomalies are marked from the video, and a distribution map of forgery risk areas is constructed based on these segments, achieving spatial localization and temporal focus of suspicious behavior. This not only provides data support for subsequent consistency verification and anomaly evolution modeling, but also effectively improves the entire identification system's ability to perceive complex forged content in the early stages, demonstrating strong practical value and technological innovation.

[0026] S002, based on the distribution map of the risk area of ​​forgery, perform semantic consistency verification operation, and check the semantic consistency of the facial micro-expressions, vocal movements and background semantics of the target person in each image frame one by one, and extract the spatial and temporal abnormal features in the corresponding frame position. To further improve the accuracy of deepfake video content recognition, particularly in identifying the consistency between human behavior and speech, and the rationality of scene semantics, a semantic consistency check based on a forgery risk area distribution map is proposed. By performing a one-to-one semantic consistency check on the facial micro-expressions, speech pronunciation, and image background semantics of the target person in the image frame, spatial and temporal anomaly features in the image frame are accurately extracted, providing a solid data foundation for subsequent forgery behavior evolution modeling. The entire implementation process consists of the following steps: Image frames identified as high-risk in the forgery risk area distribution map are read. Each frame is traversed chronologically, and the facial region of the person is located within each frame as the primary verification target. For the facial region, a ResNet-based facial landmark detection method is employed to extract 68 precise facial feature points, including specific locations such as the inner and outer corners of the eyes, the tip of the nose, both sides of the nose wings, the center of the upper lip, the edge of the lower lip, the tip of the chin, the center of the eyebrows, and the tail of the eyebrows. By analyzing the relative distance changes, angle shifts, and muscle deformation degrees between these feature points, the type and dynamic intensity of the person's facial micro-expressions (such as raising eyebrows, pursing lips, grimacing, and blinking) are calculated. These facial movement features directly reflect the person's emotional state and pre-vocal actions, forming a crucial foundation for recognizing the consistency between lip shape and pronunciation.

[0027] A facial landmark detection method based on ResNet architecture optimization refers to a deep learning method that introduces a residual connection mechanism into a traditional convolutional neural network to improve the detection accuracy and stability of facial landmarks (such as the corners of the eyes, eyebrows, corners of the mouth, and tip of the nose) in facial images. In this invention, the main function of this method is to accurately locate and geometrically analyze the facial region of the target person in each frame of a short video, providing high-quality input for subsequent micro-expression recognition and pronunciation consistency judgment. Specifically, the ResNet architecture, by stacking multiple residual blocks, can avoid gradient vanishing and training degradation problems while maintaining a relatively deep network, thereby achieving richer image feature extraction. By embedding a regression loss function for the facial region and landmark location supervision labels into this structure, the spatial distribution patterns and local texture details of different facial organs can be effectively learned. Compared to traditional shallow network or detection methods that do not use residual connections, key point detection based on ResNet structure optimization is not only more stable under occlusion, angle changes, and uneven lighting conditions, but also achieves high-precision capture of subtle muscle changes in facial expressions (such as slight eyebrow raising, corner of the mouth turning up, blinking, etc.). Therefore, it plays a crucial and fundamental role in the process of detecting the authenticity of facial behavior of people in fake videos in this invention.

[0028] For the time period corresponding to the aforementioned image frame, the corresponding speech segment in the original audio stream is loaded, and dynamic articulation behavior features of the speech are extracted. A bidirectional long short-term memory network based on Mel spectrogram input is used to model the speech sequence, extracting specific acoustic features in multiple dimensions such as speech start time, end time, syllable boundaries, intensity distribution, pitch change trajectory, and speech rate rhythm structure. Simultaneously, the opening and closing period of the mouth region image on the time axis is used for alignment analysis with the aforementioned speech intensity peak sequence to determine whether the abrupt change in speech intensity matches the timing of mouth opening and closing. If there is a situation where the mouth is still but there is continuous speech in the audio, or the mouth opens and closes rapidly but there is no significant signal in the speech, it can be determined that there is a synchronization anomaly between the articulation action and the speech trajectory, belonging to a temporal-level semantic conflict.

[0029] The Bidirectional Long Short-Term Memory (Bi-LSTM) network, based on Mel spectrogram input, is a deep learning method that uses the speech time-frequency feature map (i.e., the Mel spectrogram) transformed from Mel frequency cepstral coefficients as input. It employs a Bi-LSTM network structure to perform contextual modeling of speech sequences, extracting fine-grained acoustic features across multiple temporal dimensions. In this step, the core function of this method is to capture the dynamic changes in the speech signal, analyze its correspondence with lip movements, and thus identify whether the speech content in a fake video is natural and authentic.

[0030] The specific steps include: The original audio signal in the short video is preprocessed, including noise reduction, pre-emphasis and frame segmentation, and the continuous audio is divided into overlapping frame segments using window functions (such as Hamming window). A fast Fourier transform is performed on each frame of audio to convert it from a time-domain signal to a frequency-domain representation. The frequency-domain energy is then mapped to the Mel frequency scale that conforms to human auditory perception through a Mel filter bank, ultimately forming a two-dimensional Mel spectrum (the horizontal axis is time, the vertical axis is Mel frequency, and color or grayscale represents energy intensity). The obtained Mel spectrogram is used as input and fed into two parallel directions of the bidirectional long short-term memory network: the forward LSTM is used to learn the speech context information from the start of speech to the current frame, and the backward LSTM is used to learn the feature evolution from the end of speech to the current frame, so as to fuse the temporal semantics of the preceding and following time points at each time point and realize the modeling of dynamic attributes such as syllable boundaries, intensity changes, speech rate and rhythm. In the hidden state sequence output by Bi-LSTM, subsequent feature extraction layers are used to discriminate and quantify information in each dimension, including extracting the start and end times of each syllable, syllable duration, intensity peak position, pitch trend and frequency oscillation pattern.

[0031] Through the above modeling process, this method can achieve a comprehensive analysis of "when to pronounce", "pronunciation strength" and "speech structure rhythm" in speech content. It provides key acoustic basis for accurate alignment with pronunciation actions, mouth opening and closing timing and facial expression changes in image frames. It has significant advantages in identifying abnormal audio-visual synchronization or speech splicing artifacts in forged videos, demonstrating high practicality and creativity.

[0032] After completing the dual-channel consistency verification of facial micro-expressions and speech pronunciation, the two are further matched with the contextual semantics of the image frame. Background semantic extraction uses a semantic segmentation neural network to classify non-human areas in the image, identify the specific constituent elements of the scene, including whether it is an office environment, meeting place, street environment, or news studio, and further identify the state of other people in the background (such as standing, talking, walking), the direction and intensity of lighting (such as top light, backlight, polarized light), and whether there are semantic reference elements such as brand logos, text information, timestamps, and watermarks in the image. Then, it is judged whether the behavior of the target person matches the current scene. For example, an exaggerated smile in a meeting background, or the presence of ambient sound in the speech content in a static background without corresponding dynamic objects in the picture, are considered to be unreasonable semantic breaks between background semantics and human behavior.

[0033] The three semantic consistency verification results mentioned above are integrated and analyzed. A rule-driven and probability-based scoring strategy is used to establish a semantic consistency labeling system for each image frame, including micro-expression-pronunciation consistency scoring, pronunciation-audio track temporal matching scoring, and character behavior-scene background coordination scoring. This comprehensively forms the spatial and temporal anomaly risk level for that frame. To avoid misjudgments caused by occasional anomalies, a time-sliding window mechanism is introduced (e.g., 5 frames as a window unit). The cumulative amplitude and fluctuation frequency of the anomaly score within the window are calculated. Frame segments exhibiting continuous anomalies are merged and marked, serving as input for subsequent abnormal behavior evolution path analysis and forgery behavior tracing steps.

[0034] The core function of this step is to achieve semantic-level recognition and verification of potential abnormal behaviors in forged videos, revealing whether there are logical conflicts or temporal misalignments between visual and audio content, thereby identifying false behavioral manifestations masked by deepfake technology. In genuine short videos, facial expressions, vocalizations, and the semantics of the background should remain natural, consistent, and reasonable. For example, when a person speaks, their mouth movements should be synchronized with the rhythm of their speech, their facial expressions should match the semantics of the content being expressed, and their behavioral state should be logically consistent with the background scene. However, in deepfake videos, this consistency is often broken: for example, the person's expression may be slow or stiff, their mouth movements may be out of sync with the rhythm of their voice, or there may be unreasonable background dialogue and environmental reactions. This step addresses this issue by focusing on semantic consistency verification of the target person in the image frames based on high-susceptibility frames marked by a risk area distribution map. It analyzes the dynamic features of their facial expressions, the temporal changes in their vocalizations, and the scene adaptability of background semantic elements frame by frame, identifying features such as semantic conflicts, abnormal behavior, or temporal misalignments. By accurately extracting these spatial and temporal anomalies, it is possible to effectively identify forged video content that has high-quality image synthesis but has flaws in semantic logic. This significantly improves the overall recognition system's ability to perceive and judge complex and highly realistic forged videos, providing key evidence for subsequent evolution path modeling and forgery behavior tracing.

[0035] S003, based on spatial and temporal anomaly features, performs cross-image frame evolution feature analysis, dynamically compares and analyzes the facial expression evolution trajectory, voice frequency change trajectory and action change trajectory of the target person in the complete video time period, and constructs the temporal evolution path of the forgery behavior; To accurately analyze the temporal variation patterns of deepfake content in short videos and identify its behavioral continuity spoofing features, a cross-frame evolutionary feature analysis based on spatial and temporal anomalies is proposed. By modeling and analyzing the continuity of facial expression changes, voice frequency fluctuations, and body movement states of the target character in the video along the time axis, a clear temporal evolution path reflecting the behavioral evolution rules and anomalous mutation characteristics is constructed as the core means to identify the continuity of forged behavior and local anomalies. The overall process includes the following specific steps: In the semantic consistency verification step, image frame sequences with spatial anomalies (e.g., mismatch between mouth opening amplitude and speech pronunciation, or inconsistency between facial expressions and semantic content logic) or temporal anomalies (e.g., delayed actions, or sudden changes in expression without transition) have been identified. Based on this, the complete image frame trajectory of the target person in the video is extracted and numbered frame by frame in chronological order, continuously tracking the target person's facial region from the start frame to the end frame. This process employs optical flow tracking combined with facial keypoint matching algorithms to ensure accurate identification of the same person's position and posture in each frame under complex conditions such as camera transitions, character movement, and pose changes, providing a reliable temporal and spatial basis for subsequent feature trajectory generation.

[0036] Optical flow tracking combined with facial landmark matching is a composite tracking mechanism that combines the pixel-level motion trajectory tracking capability of optical flow in image sequences with the structural-level facial stability localization capability of facial landmark detection algorithms. This allows for the perception of detailed image changes while maintaining stable identification of the person's identity and posture. In this invention, the algorithm's role is to continuously and accurately track the facial position, orientation, and posture of the same person in each frame of a video, even during complex scene changes such as camera cuts, character movement, face rotation, and occlusion interference. This provides a consistent and temporally continuous spatial calibration foundation for subsequent facial expression evolution trajectory analysis, speech alignment analysis, and behavioral evolution modeling.

[0037] This method can be implemented by combining the existing Lucas-Kanade optical flow method with deep learning-based facial landmark detection algorithms (such as MediaPipe Face Mesh, Dlib 68 Landmark Model, or HRNet). The specific steps are as follows: In the initial frame, the facial region and the location of its key points (such as the left and right corners of the eyes, the tip of the nose, the corners of the mouth, the chin, etc.) are identified by the facial key point detection algorithm, and the two-dimensional coordinates of these key points are extracted as initial calibration points. In subsequent frames, the motion vector of each pixel or key point between adjacent frames in the image is estimated by optical flow algorithm, the position of the initial key point in the new frame is inferred, and the overall displacement direction and velocity of the face are predicted. In each new frame, the facial key point detection algorithm is reapplied to perform structural recognition of the face region, and the optical flow prediction result is compared with the current detection result. If the error is within the threshold range, the tracking is considered successful. If the deviation is large, it is corrected by historical trajectory backtracking or key point feature matching. By fusing optical flow trajectories and structural matching results, the precise position, pose angle (such as pitch angle and yaw angle), and contour information of the target person's facial region are generated in each frame, ensuring that spatial tracking of the person's identity and behavior can still be completed stably and continuously even when the person moves quickly, changes angle, or is partially occluded.

[0038] This combined approach not only overcomes the limitation of optical flow methods in drifting under structural changes, but also makes up for the problem of key point detection not being able to locate in time during high frame rates or rapid scene changes. This significantly improves the stability, robustness and accuracy of person tracking, providing solid technical support for behavioral evolution analysis and forgery identification in short videos.

[0039] After completing facial region tracking and frame sequencing, three categories of key behavioral features of the target person were extracted frame by frame. The first category was facial expression evolution features, extracting the positional changes of 68 key facial points (including eyebrows, eyelids, corners of the eyes, nose, corners of the mouth, and jaw) across consecutive frames. Expression indicators such as facial muscle stretching angles, eyelid closure ratios, and mouth corner curvature trends were calculated. The second category was speech frequency variation features, acquiring the audio segment synchronized with each frame and using Mel-frequency cepstral coefficient analysis to obtain the fundamental frequency (pitch), intensity (speech energy), speech rate (syllable density), and continuous syllable structure of the phonation portion in that frame. The third category was body movement transformation features, calculating the head center coordinates, left and right shoulder vertical angles, body balance axis angles, and head rotation direction and amplitude using a skeletal pose estimation algorithm, recording the state values ​​of the movement posture in each frame. All these features were uniformly timestamped to form a behavioral trajectory on the timeline.

[0040] The three types of behavioral feature data mentioned above are assembled into a behavioral evolution sequence in chronological order and analyzed continuously on a unified time axis. A sliding time window method (e.g., every 5 frames as a unit window) is used to calculate the smoothness, rate of change, and directional consistency of behavioral evolution segment by segment. For example, it assesses whether the angle of the corners of the mouth increases within a continuous time period, whether the frequency of eyelid closure remains regular, whether the speech pitch conforms to the natural fluctuations of the sentence structure, and whether there is a gradual trend in the left-right movement of the shoulders. If an abrupt change occurs in any behavioral channel (e.g., a sudden change in facial expression from static to exaggerated smile, an instantaneous jump in the fundamental frequency of speech, or a sudden stop in continuous movement), it is marked as a behavioral abrupt change node. If multiple channel features simultaneously exhibit asynchronous changes within the same time window (e.g., increased mouth movement but decreased speech energy, or drastic changes in facial expression but a still head), it is further marked as a cross-channel collaborative anomaly.

[0041] Based on all identified behavioral mutation nodes and collaborative anomaly segments, a temporal evolution path of forged behavior is constructed along the complete video timeline. This path uses time as the main axis and behavioral feature changes as a trajectory curve, marking the location of anomaly nodes, behavioral channel type, mutation intensity value, and state difference parameters before and after the mutation point by point. The entire path is divided into normal behavior segments and abnormal behavior segments, each visualized using different color codes to represent the transition from natural to abnormal behavior. Simultaneously, each anomaly segment in the path is accompanied by a behavioral label (such as "lip movement asynchrony," "abnormal increase in speech rate," and "abrupt head posture change"), as well as a difference index value from adjacent normal segments, facilitating backtracking analysis of forgery insertion logic and behavioral breakpoint locations. Finally, this temporal evolution path serves as an intermediate result for forgery identification and can be used in subsequent forgery tracing and dynamic identification strategy adjustment stages.

[0042] This step elevates the spatial and temporal anomalies identified in the earlier stages from isolated frame-level anomalies to continuous temporal behavioral evolution modeling. By dynamically tracking and analyzing the behavioral changes of the target person throughout the entire video segment, a temporal evolution path reflecting the occurrence, development, and spread of potential forgery is constructed. Traditional video forgery detection methods often rely solely on static judgments of single-frame images or instantaneous features of single-channel signals, making it difficult to capture forgery patterns across time scales in videos. Deep forgery content often maintains "visual continuity" across multiple frames through frame-by-frame synthesis, masking its forgery nature. Therefore, it is difficult to uncover its overall disguise logic through instantaneous analysis alone. This step addresses this challenge by extracting the facial expression change curves, speech frequency fluctuation sequences, and head or body posture change trajectories of the target person frame by frame based on the abnormal frame segments discovered in the previous semantic consistency check. Using time series modeling techniques (such as sliding window, trend analysis, and anomaly detection), it analyzes whether these behavioral features exhibit abrupt changes, nonlinear transitions, rhythmic inconsistencies, or asynchronous changes across multiple channels throughout the entire video playback process. When multiple features exhibit inconsistent trends of change within a certain timeframe, they can be marked as "forgery behavior evolution nodes." Ultimately, all anomalous evolutionary segments are integrated into a chronologically arranged "forgery behavior temporal evolution path," visually reflecting how forged content is embedded, expanded, and transformed within videos, and the range of video segments it may affect. This path not only helps pinpoint the precise starting point and behavioral variation process of forged content but also serves as foundational data support for subsequent forgery tracing, responsibility determination, and dynamic adjustment of identification strategies, playing a crucial role in improving the overall intelligence level and behavioral-level perception capabilities of the identification system.

[0043] S004, based on the time evolution path, performs forgery feature tracing operation, reverse-tracks the abnormal change points of the target person in each image frame, and identifies the position of the starting image frame and the propagation path features of the forgery behavior. To accurately identify the origin of forgery behavior in short videos and analyze its propagation and evolution over time, a forgery feature tracing method based on time evolution paths is proposed. By tracing the starting points of abnormal behavior of the target character in image frames and analyzing the distribution and expansion patterns of forgery features frame by frame on the timeline, a quantifiable forgery origin localization result and propagation path structure are ultimately formed, providing a structured foundation for subsequent identification strategy adjustment and response mechanism generation. This method includes the following specific steps: Based on the previously constructed temporal evolution path, all labeled anomalous image frames were screened and categorized, and the image frame numbers exhibiting key behavioral mutations were extracted as the initial target frame set for source tracing analysis. This set includes: image frames with drastic changes in facial expression (e.g., a sudden shift from a neutral expression to an exaggerated smile), image frames with prominent changes in speech frequency (e.g., a sudden rise in pitch exceeding 250Hz or a drop exceeding 200Hz), and image frames with obvious breaks in the trajectory of movement changes (e.g., a head orientation shifting by more than 45 degrees between adjacent frames). These anomalous frames were sorted chronologically to generate a behavioral mutation index table with timestamps, anomalous type labels, and behavioral intensity scores, which served as the input basis for the source tracing process.

[0044] Starting from the earliest anomalous frame in the behavioral mutation index table, the behavior of the target person is traced back frame by frame along the timeline, analyzing the continuity of facial expressions, speech features, and body movements in previous frames. During the tracing process, the following are monitored: whether the displacement changes of facial key points are continuous and smooth, such as whether the curvature of the corners of the mouth increases linearly between frames; whether the speech energy distribution is stable, such as whether the average speech intensity change is less than 10% in three consecutive frames; and whether the head position maintains a natural movement trajectory, such as whether the displacement vector direction and velocity of the head center point in the image coordinates are consistently consistent. When several consecutive frames meet the above stability conditions, and a drastic change in the above features occurs in the frame preceding that frame, the critical frame can be marked as the starting image frame for the generation of the forgery behavior. This frame, as the location where the forged content is first inserted or generated, has high identification value.

[0045] Starting with the identified initial frame of the forgery, the system continues to trace the forgery features of the target character in each subsequent frame along the timeline, constructing a complete propagation path for the forgery behavior. During this process, three types of feature indices are analyzed for each frame: first, facial synthesis feature indices, including changes in pixel gradients at facial edges, skin texture matching errors, and eye contour blurring; second, speech temporal anomaly indices, including mouth opening and closing rates and syllable boundary alignment deviations, and pitch jump rates between syllables; and third, motion posture continuity indices, including consistency scores for head rotation angular velocity and shoulder horizontal offset. Through these three sets of feature calculations, a forgery strength score is obtained for each frame. Frame segments with scores continuously exceeding a set threshold (e.g., 0.7) are considered high-confidence forgery segments, and these are linked together to form a propagation path trajectory covering the entire video duration.

[0046] Based on the constructed propagation path trajectory, a structured forgery tracing analysis result is generated, including four core components: first, the precise timestamp and image number of the forged starting image frame; second, the start and end range and total duration of high-risk frame segments in the propagation path; third, a curve showing the change in forgery behavior intensity, recording the trend of forgery score changes over time for each frame; and fourth, the continuity and stability score of the propagation path, calculated based on the mean difference in inter-frame behavioral features and the number of mutations. This analysis result can not only be used to generate forgery identification reports for content review, but also fed back into image recognition and audio analysis processes to dynamically adjust feature extraction parameters and recognition thresholds, achieving adaptive optimization of the identification strategy.

[0047] This step involves in-depth analysis of the temporal evolution of the target character's behavior in the video, tracing the source of the forgery from a temporal perspective, accurately identifying the starting frame of the forged content, and further revealing its propagation patterns and evolutionary characteristics throughout the video. In deepfake videos, the fake content is usually not a static segment generated all at once, but rather embedded into the real scene with high temporal consistency, especially in facial expressions, speech rhythms, or action details, exhibiting continuous characteristics that are locally natural and overall controllable. Therefore, it is necessary to use temporal modeling techniques to identify the moment when the forged content was initially inserted and analyze its propagation path, intensity, and stability. Based on the temporal evolution path generated in the previous step, this step identifies key behavioral abrupt change nodes, such as sudden and incoherent changes in facial expressions, drastic jumps in speech frequency at points without semantic transitions, or unnatural breaks in the movement trajectory. Subsequently, starting from these abnormal nodes, the process is traced back frame by frame, and the position where the forgery was first embedded is located by judging the continuity of behavioral parameters. After tracing the source, the system continues to track its propagation trajectory across multiple dimensions, including visual, audio, and motion, marking the scope of influence, evolution rhythm, and fluctuations in intensity of the forgery. This operation not only clearly visualizes the "diffusion path" of the forgery in the video but also provides fundamental support for the personalized adjustment of subsequent identification strategies and the precise deployment of public opinion response mechanisms. Through this step, the identification system can leap from "frame-level detection" to "behavioral chain recognition," playing a crucial role in dealing with highly realistic and highly concealed video forgeries, significantly enhancing the depth and practical value of intelligent identification.

[0048] S005, based on the position of the generated starting image frame and the characteristics of the propagation path, performs the recognition strategy adjustment operation, updates the image feature extraction parameters, abnormal feature recognition threshold and recognition strategy invocation method in real time, and realizes dynamic adaptation and recognition of different types of forged content; To enhance the system's ability to identify and adapt to different types of forged video content, this paper proposes an adjustment strategy based on the position of the starting image frame and the characteristics of the propagation path, addressing the differences in image features, audio trajectories, and action performance exhibited by forgery behavior. By analyzing the starting position, propagation path, intensity, and duration of the forged content in real time, the system dynamically adjusts image feature extraction parameters, anomaly detection thresholds, and the execution priority of the identification process, achieving accurate adaptation of the identification system to multiple forgery methods. The method includes the following steps: Based on the location of the initial image frame and the complete propagation path identified during the forgery feature tracing phase, the structural features of the current forgery behavior in the temporal dimension are extracted. Specifically, the extracted features include: the absolute number of the initial frame in the video sequence; the type of behavior in which the anomalous feature first appears (e.g., vocalization even when the mouth is closed, rapid head turning without accompanying background changes, sudden high-amplitude changes in facial expression); the time range covered by the propagation path (in frames); the average and maximum intensity of the anomalous behavior (in standardized scores); the density distribution of the anomalous behavior (i.e., the proportion of consecutive anomalous frames per unit time); and whether the anomalous behavior is concentrated in a single channel (image, audio, or pose) or appears simultaneously in multiple channels. Using this specific data, a feature profile of the current forgery behavior is constructed, and based on this, it is determined which known forgery type it belongs to: such as face-swapping forgery, speech synthesis forgery, or action embedding forgery.

[0049] Based on the type labels and feature profiles of the aforementioned forgery behaviors, the corresponding recognition strategy configuration file is loaded, and key parameters in the image feature extraction method are adjusted. If the forgery behavior exhibits facial face-swapping features, i.e., blurred facial edges, color block transitions, and facial expressions unrelated to semantic content appear in the starting frame, the sensitivity of Sobel edge detection in image gradient analysis is increased, while the contour region modeling strength in facial keypoint detection is enhanced, and the feature extraction window size of the facial region is expanded to capture subtle changes in the synthesized edges. If it is identified as speech synthesis forgery, i.e., multiple mouth opening and closing rhythms are inconsistent with the speech rhythm in the propagation path, the frequency segment density of the audio feature analysis part needs to be adjusted, the Mel frequency coefficient extraction dimension is increased from 20-dimensional to 40-dimensional, and the audio sampling window length is encrypted to improve the accuracy of speech rhythm recognition. If it is action embedding forgery, i.e., shoulder shifts and sudden changes in head posture angles appear near the starting frame, but the character outline does not move continuously, the weight of body keypoint stability analysis is enhanced, and the comparison threshold of the rate of change of skeletal structure between frames is increased to identify unnatural action embeddings.

[0050] Sobel edge detection is an image gradient calculation method primarily used to identify regions in an image where grayscale values ​​change significantly, i.e., the "edges" in the image, typically representing the contours of an object, structural breaks, or abrupt texture changes. In the deepfake video recognition method of this invention, Sobel edge detection is used to accurately analyze the edge details of the target person's facial region in video image frames. Its core function is to assist in discovering edge discontinuities, unnatural transitions, or abnormal texture structures caused by forgery operations. Specifically, the Sobel algorithm applies predefined convolution kernels in the horizontal and vertical directions of the image to calculate the gradient intensity and gradient direction for the region surrounding each pixel. In this way, it can detect whether facial contours (such as the jawline, cheekbone line, bridge of the nose, and hairline) exhibit suddenly enhanced edge gradients (e.g., narrowing of skin tone transition zones), local contour breaks (e.g., interruptions in lines at the corners of the mouth or eyes), or unnatural edge reconstruction (e.g., blurring or afterimages in the fusion zone) in consecutive image frames. These detailed features do not frequently appear in real faces but are typical artifacts of synthetic content. Therefore, the purpose of introducing Sobel edge detection in this step is to enhance the ability to perceive subtle synthetic traces, especially in images with high compression ratios, complex lighting, or low contrast. By using edge gradient change curves to help locate forged interpolation regions, it provides important support for the detection of image feature anomalies and effectively improves the accuracy and robustness of forgery recognition.

[0051] After updating the feature extraction parameters, the anomaly detection threshold is adjusted in real time based on the duration and intensity of the forgery behavior in the propagation path. When the forgery behavior persists for a long period in the propagation path, but the anomaly score values ​​of each frame are low and the fluctuation range is small, it is classified as "low-intensity, persistent forgery." In this case, the feature detection threshold is lowered from the default 0.75 to 0.65 to enhance the detection sensitivity under weak signal forgery. Conversely, when the forgery behavior is concentrated in a short-duration, high-intensity segment, and the anomaly score value shows an explosive growth trend, it is classified as "sudden forgery." In this case, the detection threshold needs to be increased to 0.85 to avoid misjudging real, intense behavior as forgery. At the same time, independent detection thresholds are set for audio and image channels. For example, when detecting speech splicing forgery, the audio matching tolerance value is reduced from 20 milliseconds to 10 milliseconds to accurately detect syllable breaks and pitch jumps.

[0052] Based on the analysis results of the image parameters, recognition thresholds, and forgery behavior types, the execution order and invocation method of the recognition process are dynamically configured. For face-swapping forgeries, the recognition process will prioritize facial region detection, face geometric consistency verification, facial illumination distribution comparison, and synthetic edge residual analysis. For speech synthesis forgeries, the process will prioritize audio rhythm modeling, speech spectrum integrity analysis, and lip-sync timing comparison mechanisms. For motion embedding forgeries, the process will focus on pose tracking, skeletal motion trajectory modeling, and inter-frame pose continuity detection. Finally, the above recognition processes are integrated to construct a video recognition invocation chain for the forged content, enabling the recognition system to autonomously select the optimal recognition path and judgment strategy based on the input content characteristics during operation, thereby improving the accuracy, adaptability, and response efficiency of the recognition.

[0053] This step dynamically adjusts the parameter configuration and execution process of the video recognition system based on the starting image frame position of the forgery behavior identified in the preceding analysis and its propagation path characteristics in the time dimension. This allows the system to adapt to the differences in image structure, speech trajectory, and action patterns of different types of forgery content, thereby achieving more accurate and efficient recognition results. Forged video content exhibits significant diversity in its presentation. For example, face-swapping forgeries typically show blurred edges and unnatural expressions in the facial area; speech synthesis forgeries often present abrupt changes in speech rhythm or lip movements that are out of sync with the audio; and posture manipulation forgeries may result in a lack of continuity in the character's movements between frames. Using a uniform, fixed recognition strategy to process all types of forged videos often leads to misjudgments or missed judgments due to parameter mismatches or unreasonable recognition priority settings. This step accurately identifies the type of forgery content in the current video by analyzing the specific anomaly type, the density of abnormal behaviors in the propagation path, the duration, and multimodal coupling characteristics exhibited in the starting frame of the forgery. Subsequently, the recognition system automatically updates key parameters in the image feature extraction process (such as edge detection sensitivity and facial key point coverage), adjusts thresholds for abnormal behavior recognition (such as thresholds for judging image anomaly scores or speech frequency fluctuations), and rearranges the execution order of image, audio, and pose recognition processes, prioritizing the most relevant recognition subtasks. This strategy adjustment mechanism enables the recognition system to adapt to content features, maintaining robustness and high accuracy in recognizing forged videos of varying complexity and deception methods, significantly improving its intelligence level and practical adaptability.

[0054] S006, The identification results after the identification strategy adjustment are input into the public opinion information analysis process. The public opinion monitoring response strategy is adjusted according to the identification confidence, the intensity of abnormal features and the time evolution rate to form a closed loop of identification, analysis and response, so as to achieve robust identification, self-adjusting processing and early warning control of fake public opinion information in short videos. To achieve accurate analysis and rapid response to online public opinion triggered by fake short videos, this paper proposes a method that inputs the adjusted identification results into the public opinion information analysis process and dynamically adjusts the monitoring and response strategy accordingly. This method relies on key indicators in the identification results to achieve closed-loop control throughout the entire process of identification, analysis, and response, thereby ensuring robust identification, self-regulating processing, and early warning control capabilities against fake public opinion information. The specific process includes the following steps: After completing the identification and strategy adjustment of the forgery behavior, the key structured indicators in the identification results are used as inputs to the public opinion information analysis process. The structured indicators include the following three items: (1) Identification confidence: representing the credibility score of the identification system in judging the forged content in the video, with a value range of 0 to 1. The higher the score, the more obvious the forgery features; (2) Abnormal feature intensity: including edge discontinuity score in image frames, syllable breakage index in speech trajectory, and difference score of skeleton posture change rate in human action, which are uniformly normalized to form an abnormal intensity comprehensive score; (3) Temporal evolution rate: referring to the inter-frame distance and duration change from the forgery starting image frame to the spread of abnormal behavior to various positions in the video, which is expressed as the density, average interval duration and incremental growth rate of abnormal segments on the time axis. These parameters serve as the input basis for the public opinion analysis process, driving subsequent risk assessment and strategy activation.

[0055] Upon receiving the aforementioned indicators, the public opinion information analysis process first performs an initial risk assessment of the short video content based on the identification confidence level. If the identification confidence level is higher than 0.85 and the anomalous feature strength exceeds 0.7, the video is marked as a "high-risk candidate for fake content" and enters the priority processing channel. Simultaneously, it retrieves the video's dissemination data on short video platforms, including views (daily cumulative views), comments (total user interaction messages), reposts (including in-platform reposts and external link copying), likes (positive sentiment feedback explicitly marked by the platform), and the publishing account's historical behavior records (such as the number of videos posted in the past 30 days, content types, and matching with violation tags). By correlating the above dissemination behavior data with the identification output through modeling, it assesses whether the video has triggered cross-platform dissemination, whether it has impacted specific sensitive events or group issues, and forms a dissemination impact level assessment result accordingly.

[0056] Based on the assessment results of the dissemination impact level, a dynamic adjustment mechanism for the public opinion monitoring response strategy is initiated. The adjustment operation includes the following three aspects: (1) Content review frequency adjustment: For fake videos with a dissemination level of "highly sensitive", a review polling interval of no more than 5 minutes is set to ensure that the subsequent dissemination status and user feedback of the video can be updated in minutes; (2) Activation of response behavior strategy: If the video reaches the "moderate dissemination impact" standard on multiple platforms, an early warning response behavior is triggered, and platform-side processing suggestions are generated and pushed, including implementing the "visible only to private messages" restriction on the video, prominently marking the prompt "may contain false information" at the top of the playback interface, and temporarily blocking the comment function until the manual review is completed; (3) Account behavior archiving mechanism: The user account that published the video is included in the dynamic monitoring list of high-risk accounts, the identification results of all recent videos are extracted, the proportion of fake content published, the dissemination speed and content similarity are calculated, and the content publishing risk curve of the account is constructed.

[0057] The monitoring and response execution status and subsequent user behavior feedback results are used as closed-loop feedback data input to the identification process for strategy optimization. Specific feedback content includes: whether platform actions were successfully triggered, whether the video was removed or had its reach limited, whether public opinion feedback stabilized due to the response (e.g., a decrease in the frequency of emotionally polarized words in the comments section), and the trends in user reports and likes. Based on this feedback, the weight parameters of each feature channel in the next round of identification are automatically adjusted. For example, sensitivity to short-term frequency changes is increased in the voice channel detection, and the gradient gain threshold for edge fusion detection is increased in the image channel detection. Simultaneously, the priority of identification strategy calls is updated, prioritizing the execution of strategy chains proven effective in this round of feedback, ultimately achieving a continuous linkage process of "forgery identification—public opinion analysis—dynamic response—closed-loop feedback—identification re-optimization."

[0058] The core function of this step is to use the analysis results obtained in the preceding forgery identification stage as a data foundation to further guide the precise execution of the public opinion information analysis process. Based on this, it enables dynamic monitoring and response strategy adjustment for online public opinion risks, constructing a closed-loop operation mechanism of "identification-analysis-response-feedback." Its key significance lies in using the identification information of forged content not only for judging the content itself but also as a driving variable for public opinion governance, strengthening the comprehensive assessment capability of video dissemination paths, impact scope, and potential risks. Specifically, identification confidence reflects the credibility of the identification system's judgment on a certain video forgery behavior; anomaly feature strength measures the comprehensive expression of the degree of disturbance of forged content in image, audio, or behavioral dimensions; and time evolution rate reflects the rhythm and diffusion efficiency of the forgery behavior in the short video dissemination process. Once these three data indicators are input into the public opinion analysis process, a multi-dimensional risk profile can be constructed to determine whether the video has the potential to trigger online emotional fluctuations, mislead public perception, or escalate sensitive social issues. Based on this, public opinion monitoring and response strategies can be adjusted in real time, including increasing the frequency of review, prioritizing the triggering of early warning mechanisms, adjusting content display strategies, recording the risky behavior of publishing accounts, and restricting the visibility of their content. More importantly, this adjustment is not a one-time static process, but rather forms a cyclical feedback mechanism with the identification results: the public opinion analysis results can, in turn, be used as parameters input into the identification model to optimize the next round of identification weight allocation and judgment strategy invocation, thereby achieving adaptive adjustment and continuous optimization in complex propagation environments. Through this step, a comprehensive capability is achieved for sensitivity identification, propagation pattern analysis, strategy-level response, and model closed-loop self-evolution of online public opinion events triggered by deepfake videos, significantly improving the intelligence level and robustness of short video content security management.

[0059] The proposed image processing-based short video online public opinion information identification method effectively addresses the shortcomings in identifying deepfake video content, achieving accurate identification and dynamic response to multimodal and cross-temporal forgery features. This method integrates image frames, audio trajectories, and temporal sequence information to construct a multimodal joint analysis model, ensuring comprehensive extraction of forgery clues even in highly complex and structurally varied short video environments. Furthermore, semantic consistency checks and cross-frame evolutionary modeling identify subtle anomalies in facial expressions, voice frequencies, and gestures, enhancing the sensitivity to detecting concealed forgery. Based on this, the method uses the forgery origin frame and propagation path to trace the source, improving the interpretability and accuracy of forgery content tracing. Finally, through dynamic identification strategy adjustment and closed-loop control of the public opinion response mechanism, it achieves intelligent and adaptive processing throughout the entire process from identification to handling. This method not only improves the accuracy and robustness of identification but also enhances the real-time nature and controllability of public opinion governance, providing technical support for platform-level video review, online content supervision, and emergency response, demonstrating significant engineering application value and social governance benefits.

[0060] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for identifying short video network public opinion information based on image processing technology, characterized in that, The method comprises the following steps: S001, by acquiring the image frames, audio tracks and time sequence information of the short video, constructing a multi-modal fusion model, extracting continuous image frames with suspicious identity features, and generating a forgery risk area distribution map; S002, according to the forgery risk area distribution map, performing semantic consistency verification, extracting spatial and temporal abnormal features existing between the facial micro-expression, pronunciation action and background semantics in the image frame; S003, based on the spatial and temporal abnormal features, performing cross-frame evolution analysis, and constructing the time evolution path of the target person's expression, voice and action track; S004, according to the time evolution path, performing forgery feature tracing, identifying the generation starting image frame position and propagation path features of the forgery behavior; S005, according to the starting image frame position and propagation path features, performing identification strategy adjustment, dynamically updating the image feature extraction parameters and identification threshold, and realizing adaptive identification of the forgery type; S006, input the identification result after identification strategy adjustment into the public opinion analysis process, adjust the monitoring response strategy according to the identification confidence and evolution rate, and form a closed loop mechanism of identification, analysis and response.

2. The short video network public opinion information identification method based on image processing technology according to claim 1, characterized in that, Step S001 comprises: Decode the short video frame by frame by fixing the frame rate, extract the image frame sequence and save it in RGB three-channel format, and keep the original resolution; Extract the speech signal by using the audio stream analysis method, and construct the audio time sequence vector based on the 16kHz sampling rate, extract the mel frequency cepstrum coefficient, tone fundamental frequency, short-time energy spectrum, syllable duration, speech rhythm curve, sound intensity change value and formant distribution features; According to the time stamp of the image frame and the audio segment, the time sequence mapping relationship is established, and the time synchronization relationship between the image frame and the audio segment is formed; Based on the visual features of the image frame, the audio track and the time sequence information, a multi-modal fusion model is constructed, the multi-modal joint features are analyzed, the continuous image frame segments with suspicious identity features are extracted, and the forgery risk area distribution map is generated according to the semantic anomaly. 3.The short video network public opinion information identification method based on image processing technology according to claim 1, characterized in that, Step S002 comprises: Based on the forgery risk area distribution map, read the high-risk image frame segment, and locate the face region of the person; Using the face key point detection method based on residual neural network structure optimization, the face feature points are extracted and the face micro-expression state is analyzed; Convert the corresponding time period audio signal to Mel spectrogram, input it into bidirectional long short-term memory network to extract audio features of pronunciation action, and analyze the consistency with mouth opening and closing action alignment; Extract the image background semantic information, judge whether there is semantic conflict between the person's behavior and the scene composition, and fuse the three types of verification results to form the spatial and temporal abnormal risk level of the image frame.

4. The short video network public opinion information identification method based on image processing technology according to claim 1, characterized in that, Step S003 comprises: According to the spatial abnormality and time abnormality image frame, the target person image frame sequence is extracted, and the face region is continuously tracked by using optical flow tracking combined with face key point matching algorithm; Extract the facial expression features, speech frequency features and action posture features in each frame, and perform unified timestamp labeling; Based on the sliding time window, the continuity index of the behavior features on the time axis is calculated, and the abnormal mutation nodes and cross-channel collaborative abnormalities are identified; The behavior time evolution path is constructed in the time dimension, and the mutation node position, feature type and difference parameter are marked to identify the abnormal evolution process of the fake behavior.

5. The short video network public opinion information identification method based on image processing technology according to claim 1, characterized in that, Step S004 includes: According to the time evolution path, the key behavior mutation image frame is extracted, and a behavior mutation index table containing time stamp, abnormal type label and behavior intensity score is generated; Starting from the earliest abnormal image frame, the target person behavior state is traced back frame by frame to the front of the time axis, and the last frame that meets the continuous stability condition is identified as the generation starting image frame of the fake behavior; Taking the generation starting image frame as the starting point, the fake feature performance is tracked forward to the time axis, and the fake propagation path trajectory covering the whole video period is constructed; Based on the fake propagation path trajectory, the fake traceability analysis result is generated, including the starting image frame timestamp, propagation path range, fake behavior intensity change curve and propagation path stability score.

6. The short video network public opinion information identification method based on image processing technology according to claim 1, characterized in that, Step S005 includes: Extracting the fake starting image frame position and propagation path trajectory, analyzing the abnormal behavior type, time range, behavior intensity and abnormal behavior distribution channel; Based on the abnormal behavior type and feature portrait, the image feature extraction parameter and audio modeling strategy are adjusted to improve the feature recognition accuracy of face swapping, voice synthesis and action embedding forgery; According to the duration and intensity change trend of the fake behavior, the image recognition threshold and audio recognition threshold are dynamically updated to realize adaptive recognition of different fake intensity types; According to the fake behavior type, the recognition process calling sequence is configured to construct the recognition process calling chain, so as to realize accurate adaptation and real-time recognition of different types of fake content.

7. The short video network public opinion information identification method based on image processing technology according to claim 1, characterized in that, Step S006 includes: The recognition confidence, abnormal feature intensity and time evolution rate are input into the public opinion information analysis process as structured indicators; According to the recognition confidence and abnormal feature intensity, the risk level of fake content is judged, and the propagation influence level evaluation result is constructed combined with the propagation behavior data; According to the propagation influence level, the content review frequency adjustment, response behavior strategy activation and account behavior archiving mechanism are executed; The execution result and user feedback are input into the recognition process to optimize the weight parameter and update the strategy calling priority, forming a recognition, analysis and response closed loop.

8. The short video network public opinion information identification method based on image processing technology according to claim 7, characterized in that, The propagation influence level evaluation result construction step includes: Obtain the propagation behavior data of the target short video on multiple platforms, including daily play times, comment quantity, forwarding times, like number and historical behavior records of the publishing account; Joint modeling of propagation behavior data, recognition confidence, abnormal feature intensity and time evolution rate is performed to identify whether there is cross-platform propagation, high-frequency interaction behavior or sensitive event involved; According to the propagation breadth, propagation speed and interaction intensity, the propagation risk score is calculated, and divided into three propagation influence levels: high sensitivity, medium sensitivity and low sensitivity.

Images