Method, system and computer storage medium for identifying negative information

Abstract

The present application relates to the technical field of data processing, and relates to a negative information recognition method and system and a computer storage medium, comprising: performing text conversion on audio data in an acquired video to be detected to obtain target text; performing sentiment analysis and negative recognition processing on the target text to obtain an audio recognition result carrying a frame extraction review label; determining a corresponding suspected video range set in the video to be detected according to the audio recognition result carrying the frame extraction review label; performing frame extraction operation on the suspected video range set using a preset frame extraction strategy to obtain a key frame image set corresponding to the suspected video range; performing visual analysis on the key frame image set to recognize negative content of the key frame image set to obtain a target visual review result corresponding to the key frame image set; and performing fusion judgment on the audio recognition result and the target visual review result based on a preset fusion strategy to obtain a target judgment result for the video to be detected.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method, system, and computer storage medium for identifying negative information. Background Technology

[0002] With the rapid development of internet technology and the popularization of short video platforms, user-generated content (UGC) has experienced explosive growth. A large amount of negative and illegal information, such as false advertising, malicious attacks, and the spread of rumors, is mixed in with this massive amount of video content, which seriously affects the health of the online ecosystem and user experience.

[0003] Traditional negative content recognition technologies primarily rely on single-modal analysis. For example, methods based on single-audio analysis convert speech to text using Automatic Speech Recognition (ASR), then perform sentiment analysis and negative keyword matching to determine negative content. Alternatively, methods based on single-image analysis extract frames from a video at fixed intervals, using OCR text extraction and content understanding to identify negative information in the video. However, these methods often suffer from insufficient detection coverage and are prone to missed detections when faced with audio-visual asymmetry scenarios (such as negative information appearing only in the audio while the video is normal, or negative text appearing only in the video while the audio is irrelevant). This is often due to incomplete information sources or intermodal information conflicts.

[0004] Therefore, how to integrate multimodal information to overcome the limitations of single-modal analysis has become a pressing technical problem in the field of video negative content recognition. Summary of the Invention

[0005] One objective of this invention is to provide a method, system, and computer storage medium for identifying negative information, in order to solve the technical problem of insufficient detection coverage / easy missed detection in the face of audio-visual asymmetry scenarios in the prior art.

[0006] In a first aspect, embodiments of the present invention provide a method for identifying negative information, the method comprising: The audio data in the acquired video to be detected is converted into text to obtain the target text; Sentiment analysis and negative recognition processing are performed on the target text to obtain audio recognition results carrying tags that need to be extracted for verification; Based on the audio recognition results carrying the tags that need to be reviewed by frame extraction, a set of suspected video ranges is determined in the video to be detected; A preset frame extraction strategy is used to perform frame extraction on the suspected video range set to obtain a set of keyframe images corresponding to the suspected video range. Visual analysis is performed on the keyframe image set to identify negative content in the keyframe image set and obtain the target visual verification result corresponding to the keyframe image set. The audio recognition result and the target visual verification result are fused and judged based on a preset fusion strategy to obtain a target judgment result for the video to be detected. The target judgment result is used to indicate whether the video to be detected contains negative information.

[0007] In a second aspect, a negative information identification system is provided, the negative information identification system including an electronic device, the electronic device including a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor, when executing the one or more computer programs, causing the negative information identification system to implement the negative information identification method as described in the first aspect.

[0008] In a third aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the negative information identification method as described in the first aspect.

[0009] In a fourth aspect, a negative information identification device is provided, the negative information identification device comprising: The processing unit is used to convert the audio data in the acquired video to be detected into text to obtain the target text; The processing unit is also used to perform sentiment analysis and negative recognition processing on the target text to obtain audio recognition results carrying tags that need to be extracted for verification. The processing unit is further configured to determine a set of suspected video ranges in the video to be detected based on the audio recognition results carrying the frame-sampling verification tags. The processing unit is further configured to perform frame extraction operation on the suspected video range set using a preset frame extraction strategy to obtain a set of keyframe images corresponding to the suspected video range. The analysis unit is used to perform visual analysis on the keyframe image set, identify negative content in the keyframe image set, and obtain the target visual verification result corresponding to the keyframe image set. The analysis unit is further configured to perform a fusion judgment on the audio recognition result and the target visual verification result based on a preset fusion strategy to obtain a target judgment result for the video to be detected. The target judgment result is used to indicate whether the video to be detected is a video containing negative information.

[0010] Compared with existing technologies, in the embodiments implemented by the above-mentioned negative information identification method, negative information identification system, negative information identification device, and computer storage medium, this embodiment first performs text conversion and emotion and negative identification on the audio, determines the suspected video range only for audio identification results carrying the label of needing frame extraction verification, and performs frame extraction only within the set of suspected video ranges using a preset frame extraction strategy for subsequent visual analysis. This avoids full frame extraction and full visual analysis of the video to be detected, thereby reducing the consumption of computing resources and improving the overall processing efficiency of the system. At the same time, negative content identification is performed on the key frame image set within the suspected range through visual analysis to obtain the target visual verification result. Then, based on a preset fusion strategy, the audio identification result and the visual verification result are fused and judged, realizing joint decision-making of audio and visual dual-modal information, effectively overcoming the limitations of single-modal analysis, and improving the accuracy and robustness of negative content identification. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of a negative information recognition system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for identifying negative information according to an embodiment of the present invention; Figure 3 This is a schematic diagram showing the distribution of frame extraction time points within the time range of the segment to be reviewed for the three preset frame extraction strategies in this embodiment of the invention. Figure 4 This is a schematic diagram of the structure of a negative information recognition device according to an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0014] It should be noted that, unless otherwise specified, the various features in the embodiments of this invention can be combined with each other, all of which are within the protection scope of this invention. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this invention do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0015] In existing real-world scenarios, a large amount of negative content exhibits the characteristic of "audio-visual asymmetry," that is, the audio content is negative but the visual is normal, or the visual displays negative information but the audio is normal. Single audio analysis methods cannot detect content with "normal audio and negative visuals," while single image analysis methods are prone to missing content with "negative audio and normal visuals." Both have the defect of insufficient detection coverage.

[0016] Therefore, multimodal fusion technology has gradually become a research hotspot. Existing multimodal fusion methods usually extract audio and image features simultaneously and use simple strategies such as feature concatenation and classification, multimodal feature weighted fusion, or attention-based feature fusion to perform fusion at the feature layer or decision layer.

[0017] However, existing multimodal fusion methods lack a dedicated processing mechanism for "audio-video asymmetry" scenarios, resulting in extremely low recognition accuracy for such content. Furthermore, they lack precise frame extraction strategies, performing high-density full frame extraction on all videos uniformly. Moreover, they lack a layered filtering mechanism during the analysis phase, failing to perform full, undifferentiated analysis. This forces a large amount of normal content to undergo complex processing procedures, leading to a serious waste of computing resources and low overall system processing efficiency.

[0018] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a negative information identification system. Figure 1 In the negative information identification system 10, there is an electronic device 20, which includes at least one processor 201 and a memory 202.

[0019] Among them, electronic device 20 can be a server with audio and video data processing and multimodal fusion capabilities, a dedicated content review device, an edge computing device, a cloud computing node, or an intelligent network device (such as an intelligent router or switch) that integrates automatic speech recognition (ASR) and visual analysis modules, etc., without being limited to a single type.

[0020] The processor 201 is configured to support the negative information recognition system in performing the corresponding functions of the negative information recognition method in the above method embodiments. The processor 201 can be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0021] The memory 202 is used to store program code and storage components such as CommonStorage, graph databases, vector databases, MySQL, Redis, and MQ, ensuring data storage and management. The memory 202 may include volatile memory (VM), such as random access memory (RAM); it may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may include a combination of the above types of memory.

[0022] See Figure 2 , Figure 2 A flowchart illustrating a method for identifying negative information provided in an embodiment of the present invention includes the following steps: S10. Convert the audio data in the acquired video to be detected into text to obtain the target text.

[0023] The video to be detected refers to a short video or video stream that needs to be identified for negative content. It contains audio signals and video images. In other words, the video to be detected is a mixed audio and video data file that needs to be determined to contain illegal or negative information.

[0024] Audio data refers to the sound signals contained in the video, which are usually digital audio streams and include speech sounds, ambient sound effects, etc.

[0025] Text-to-text conversion is the process of converting speech information in audio into corresponding text information, which can be referred to as Automatic Speech Recognition (ASR).

[0026] The target text refers to text data that carries timestamp information after being converted from speech recognition.

[0027] In specific implementation, a video file to be detected is acquired, and an independent audio data stream is separated from the video. The audio data stream is then input into a deep learning automatic speech recognition model (e.g., Whisper automatic speech recognition model or Qianwen automatic speech recognition model). The deep learning automatic speech recognition model performs acoustic feature extraction on the audio data stream, converting the audio data into an acoustic feature vector sequence. The decoder then maps the acoustic feature vector sequence into a corresponding text character sequence, while generating timestamp information corresponding to each text character sequence. The text character sequence and the corresponding timestamp information are concatenated and combined to generate a target text sequence containing timestamp information, which is the target text.

[0028] Among them, deep learning automatic speech recognition models refer to algorithmic models trained on deep neural networks that are used to convert human speech signals into corresponding text content, such as the Whisper automatic speech recognition model.

[0029] The timestamp information refers to the data that records the time position of a specific text fragment in the original video to be detected, including the start time node and the end time node.

[0030] For example, suppose the video to be tested is a 1-minute UGC short video containing user reviews of a product; after extracting the audio data, the audio preprocessing module performs noise suppression. Subsequently, the Qianwen ASR automatic speech recognition system is used to convert the audio into text, outputting the target text "This product is really bad, the quality is poor", with the corresponding timestamp from 0 minutes 15 seconds to 0 minutes 20 seconds.

[0031] As can be seen, in this embodiment, the target text is obtained by converting the audio data in the video to be detected into text, which enables text operations to replace audio waveform operations, reducing the complexity of data parsing; and the target text form obtained after conversion allows for direct string comparison and semantic calculation based on the text, improving the efficiency of data processing.

[0032] S20. Perform sentiment analysis and negative recognition processing on the target text to obtain audio recognition results carrying tags that need to be extracted for verification.

[0033] Sentiment analysis refers to the use of natural language processing technology to determine the emotional tendency expressed in a target text. It is usually divided into three categories: positive, neutral, and negative, and is used to identify the emotional nature conveyed by the text.

[0034] Negative identification processing refers to the process of detecting and labeling potential negative content in text. It can determine whether it contains risky information by combining negative keyword matching and sentiment analysis results.

[0035] In one embodiment, the step of performing sentiment analysis and negative recognition processing on the target text to obtain an audio recognition result carrying a tag requiring frame extraction and verification includes: S201. Perform sentiment analysis on the target text to obtain the sentiment polarity result of the target text.

[0036] In practice, the target text is input into a deep learning model (such as BERT or LSTM networks) or a rule engine; further, the sentiment tendency of each subtext in the target text is determined, and the sentiment polarity label is output, which includes "positive", "neutral" and "negative"; and combined with the context and syntactic structure, the overall and local sentiment tendency of the text is comprehensively judged to obtain the sentiment polarity result corresponding to each subtext.

[0037] S202. Using multiple subtexts in the target text as query identifiers, perform negative keyword matching in a preset negative keyword database, and calculate the negative keyword matching degree corresponding to each subtext based on character overlap.

[0038] Subtext refers to the target text being divided into several smaller text units, such as sentences, phrases, or paragraphs, to facilitate analysis and judgment one by one.

[0039] Optionally, the subtext can be obtained by segmenting the target text according to timestamp information or sentence pauses.

[0040] The query identifier refers to a specific string used in the database retrieval process to compare and match entries stored in the database.

[0041] The preset negative keyword database is a pre-built thesaurus containing keywords or phrases related to negative content, used to match corresponding words in the target text.

[0042] Character overlap is an indicator that measures the degree of matching between subtext and negative keywords at the character level, reflecting the similarity between the text and the negative keywords.

[0043] The character overlap can be obtained from, but is not limited to, character matching ratio, edit distance, or word frequency statistics.

[0044] Among them, negative keyword matching degree refers to the indicator that measures the degree of matching between subtext and negative keywords at the character level, reflecting the similarity between text and negative keywords.

[0045] In practice, the target text can be divided into multiple sub-texts based on punctuation marks or timestamp intervals; each sub-text is used as a query identifier, and candidate keywords whose character length difference with the query identifier is within a preset range are retrieved from a preset negative keyword database; the proportion of the number of identical characters between the query identifier and the candidate keywords is calculated to the total number of characters in the query identifier, and this proportion is determined as the negative keyword matching degree corresponding to the sub-text.

[0046] S203. Based on the negative keyword matching degree corresponding to each sub-text and the sentiment polarity result of the target text, obtain the audio recognition result carrying the label that needs to be extracted for verification.

[0047] The specific implementation of S203 includes the following steps S2101-2105.

[0048] S2101. When the negative keyword matching degree is greater than or equal to a preset matching degree threshold, the corresponding subtext is determined to be a suspected risk subtext; or, S2102. When the negative keyword matching degree is less than the preset matching degree threshold, the corresponding subtext is determined to be a non-reviewed subtext. S2103. When the sentiment polarity result corresponding to the suspected risk sub-text is negative, the suspected risk sub-text is determined as a sub-text to be reviewed; or, S2104. When the sentiment polarity result corresponding to the suspected risk subtext is non-negative, the corresponding subtext is determined to be a non-reviewable subtext.

[0049] S2105. The sub-text to be reviewed in the target text is marked to obtain the audio recognition result carrying the tag that needs to be framed for review.

[0050] The preset matching threshold is a preset limit value used to determine whether the subtext matches negative keywords sufficiently. If it exceeds this value, it is judged as suspected negative, that is, suspected risk subtext.

[0051] Therefore, suspected risky subtext refers to subtext where the negative keyword matching degree reaches or exceeds the preset matching degree threshold.

[0052] Among them, the subtext that is not reviewed refers to the subtext whose negative keyword matching degree does not reach the preset matching degree threshold, or whose sentiment polarity result is non-negative.

[0053] Among them, the subtext to be reviewed refers to the subtext that simultaneously meets the requirements of negative keyword matching degree reaching the preset matching degree threshold and sentiment polarity result being negative.

[0054] Among them, the frame extraction verification label refers to the marking information added to the sub-text to be verified, which is used to instruct the subsequent device to perform image frame extraction operation on the corresponding time period of the sub-text.

[0055] Among them, the audio recognition results carrying the frame-extraction verification label refer to the structured data set of the target text carrying the frame-extraction verification label and its corresponding negative keyword matching degree, sentiment polarity result and frame-extraction verification label after the dual judgment of sentiment analysis and negative keyword matching.

[0056] In the specific implementation of S2101, the negative keyword matching degree corresponding to each subtext is compared with the preset matching degree threshold; if the negative keyword matching degree is greater than or equal to the preset matching degree threshold, it is determined that the subtext has triggered the lexical risk condition and the subtext is identified as a suspected risk subtext.

[0057] In S2102, if the negative keyword matching degree is less than the preset matching degree threshold, the negative content recognition system determines that the subtext has not triggered the word risk condition and identifies the subtext as a non-reviewable subtext.

[0058] In S2103, the sentiment polarity result corresponding to the suspected risk subtext is extracted; if the sentiment polarity result is negative, the suspected risk subtext is determined to meet both the lexical risk condition and the semantic risk condition, and the suspected risk subtext is identified as a subtext to be reviewed.

[0059] In S2104, if the sentiment polarity result corresponding to the suspected risky subtext is positive or neutral (i.e., not negative), it is determined that although the subtext has lexical risk, its semantic tendency is normal, and the subtext is identified as a subtext not to be reviewed.

[0060] In S2105, a frame-extraction verification label is added to all sub-texts to be verified, and the sub-texts to be verified with the added labels are combined with the sub-texts not to be verified in the order of the original target text to generate an audio recognition result carrying the frame-extraction verification label.

[0061] For example, suppose the target text is: "The quality of this product is very poor, and the after-sales service is terrible." Subtext 1: "The quality of this product is very poor," and subtext 2: "The after-sales service is terrible." Both subtexts are determined to have negative sentiment. The negative keyword database contains words such as "poor" and "terrible." Character overlap calculations show that the matching degree between the two sentences and the negative keywords exceeds a threshold (e.g., 0.7). Therefore, according to S2101 and S2103, both subtexts are marked as subtexts to be reviewed and tagged with "requires frame extraction review," indicating that the corresponding video segments need to undergo video frame extraction and visual review.

[0062] As can be seen, this embodiment, by performing fine-grained sentiment analysis and negative keyword matching on the target text, combined with a matching degree threshold determination mechanism, can accurately identify text fragments with potential negative risks and clearly negative sentiment attitudes, and selectively label content that needs visual review. Therefore, it improves the detection recall rate of negative content, while avoiding unnecessary processing of risk-free text, promoting the efficient execution of subsequent intelligent frame extraction and multimodal fusion judgment, and improving the accuracy and processing efficiency of the negative information identification method.

[0063] S30. Based on the audio recognition results carrying the tags that need to be extracted for verification, determine the corresponding set of suspected video ranges in the video to be detected.

[0064] Optionally, determining the corresponding set of suspected video ranges in the video to be detected based on the audio recognition results carrying the tags for frame extraction and verification includes: parsing the audio recognition results carrying the tags for frame extraction and verification to obtain multiple sub-texts to be verified; obtaining the time range corresponding to each sub-text to be verified, the time range including start time and end time; extracting segments from the video to be detected based on the time range corresponding to each sub-text to be verified to obtain segment data to be verified corresponding to each sub-text to be verified; and summarizing all segment data to be verified to obtain the corresponding set of suspected video ranges.

[0065] Among them, the subtext to be reviewed refers to the text segment marked as needing frame-by-frame review in the audio recognition results, which represents the content in the audio that is judged to be suspected of being negative.

[0066] The time range refers to the time interval corresponding to the sub-text to be reviewed in the original video to be detected, which is defined by the start time and the end time.

[0067] The start time refers to the absolute time point at which the audio segment corresponding to the sub-text to be reviewed begins in the video to be tested. The end time refers to the absolute time point at which the audio segment corresponding to the sub-text to be reviewed ends in the video to be tested.

[0068] Among them, segment extraction refers to extracting corresponding continuous video segment data from the video file to be detected according to the time range.

[0069] Among them, the data to be reviewed refers to the video segments extracted from the video to be tested, which correspond to the time range of the sub-text to be reviewed, and include image frames and audio data within that time period.

[0070] The suspected video range set refers to the set of data from all segments of the video to be tested that are marked as suspected negative content and are to be reviewed.

[0071] In the specific implementation of parsing the audio recognition result carrying the frame extraction verification label to obtain multiple sub-texts to be verified, the input audio recognition result carrying the frame extraction verification label is subjected to data parsing processing, the label field in the audio recognition result is traversed, all text units carrying the frame extraction verification label are filtered out, and the above text units are extracted to obtain multiple sub-texts to be verified.

[0072] For example, if the audio recognition results contain the text "What car crashed?" and "broken axle" marked as needing frame extraction for review, the system will parse them out as sub-texts to be reviewed.

[0073] In the specific implementation of obtaining the time range corresponding to each sub-text to be reviewed, which includes the start time and the end time, according to the mapping relationship between the sub-text to be reviewed and the timestamp, the timestamp information associated with each sub-text to be reviewed is read, the corresponding time start point is extracted from the timestamp information as the start time, and the corresponding time end point is extracted as the end time. The start time and the end time constitute the time range corresponding to the sub-text to be reviewed.

[0074] For example, in the time range corresponding to the subtext "What car crashed?" to be reviewed, the start time is 1 minute 12 seconds 01 and the end time is 1 minute 13 seconds 25; the start time corresponding to the subtext "Broken axle" to be reviewed is 2 minutes 21 seconds 41 and the end time is 2 minutes 22 seconds 03.

[0075] In the specific implementation of extracting segments from the video to be tested according to the time range corresponding to each sub-text to be tested, the start time and end time corresponding to each sub-text to be tested are used as boundary conditions. The video data is read from the start time until the end time. The continuous video data read is extracted as the segment data to be tested corresponding to the sub-text to be tested.

[0076] For example, video data between 1 minute 12 seconds 01 and 1 minute 13 seconds 25 is extracted from the video to be tested as the segment data to be reviewed corresponding to "What car crashed?"; video data between 2 minutes 21 seconds 41 and 2 minutes 22 seconds 03 is extracted as the segment data to be reviewed corresponding to "Broken axle".

[0077] In the specific implementation of summarizing all the data of the segments to be reviewed and obtaining the corresponding set of suspected video ranges, all the extracted data of the segments to be reviewed are arranged and combined according to the chronological order or the order in which the subtext appears in the target text. All the combined data of the segments to be reviewed are integrated into a data set, and this data set is determined as the set of suspected video ranges. This set only contains video segments that are initially judged to be suspected negative, and excludes normal video segments that have not been marked.

[0078] As can be seen, this embodiment accurately obtains the time interval of each suspected negative audio segment in the video by parsing the audio recognition results carrying the tags that need to be extracted for frame review, and extracts the corresponding video segments accordingly, thereby achieving accurate positioning and extraction of suspected negative content video areas; effectively avoiding blindly extracting frames from the entire video, optimizing the utilization of computing resources, improving the system's processing efficiency and detection accuracy, and providing accurate inspection objects for subsequent visual review, thus enhancing the comprehensiveness and robustness of negative information recognition.

[0079] S40. A preset frame extraction strategy is used to perform frame extraction on the suspected video range set to obtain a set of keyframe images corresponding to the suspected video range.

[0080] Frame extraction refers to the process of extracting single-frame images from a video stream or video file according to certain rules and time points. Video is composed of consecutive frames (images). Frame extraction allows for the capture of specific frames at designated times, obtaining key visual content for subsequent analysis. The purpose of frame extraction is to select representative frames from large amounts of video data, reducing processing load while ensuring the integrity and validity of the information.

[0081] The keyframe image set refers to the aggregated set of all key image frames extracted after performing frame extraction operations on the suspected video range set.

[0082] In one embodiment, the step of performing frame extraction on the suspected video range set using a preset frame extraction strategy to obtain a set of keyframe images corresponding to the suspected video range includes: S401. Obtain the video duration corresponding to the video to be detected.

[0083] Among them, video duration refers to the total length of time from the start to the end of the playback of the video to be tested.

[0084] Specifically, the metadata information of the video to be detected is read, and the value representing the total length of the video playback is parsed from the metadata information. This value is then determined as the video duration corresponding to the video to be detected.

[0085] S402. Determine the number of segments to be reviewed based on the video duration.

[0086] The number of segments to be reviewed refers to the maximum number of segments allowed to perform frame extraction operations, determined based on the video duration.

[0087] Specifically, the number of segments to be reviewed in the entire video can be calculated based on preset time segment length thresholds or calculation rules; for example, every 5 minutes can be considered as a segment, or the number of segments can be dynamically adjusted according to the density of negative content.

[0088] Optionally, a mapping table between preset video duration and the number of segments can be obtained, and the obtained video duration can be matched with the mapping table to find the corresponding quantity value; or the quantity value can be calculated by dividing the video duration by a preset unit time length, and this quantity value can be determined as the number of segments to be reviewed. The longer the video duration, the more segments to be reviewed can be determined.

[0089] S403. Analyze the set of suspected video ranges to obtain multiple sets of segment data to be reviewed.

[0090] Among them, the segment data to be reviewed refers to the video segments extracted from the video to be tested, which correspond to the time range of the sub-text to be reviewed.

[0091] Specifically, the set of suspected video segments is unpacked, the elements in the set are traversed, and the data of each video segment in the set is extracted to obtain multiple sets of segment data to be reviewed.

[0092] S404. Based on preset selection rules, select a target subset of segments to be reviewed from the multiple sets of segments to be reviewed, corresponding to the number of segments to be reviewed.

[0093] The target subset of segments to be reviewed refers to the set of segments selected from multiple sets of segments to be reviewed, whose number corresponds to the number of segments to be reviewed, and which actually performs frame extraction operations.

[0094] The preset selection rule refers to the criteria for selecting the segments from multiple sets of data to be reviewed to actually perform frame extraction. The preset selection rule may include, but is not limited to, selection rules based on chronological order, selection rules based on risk confidence, or selection rules based on uniform temporal distribution; no single limitation is made here.

[0095] Specifically, the selection rule based on chronological order is to sort the data segments to be reviewed according to the order in which their start times appear in the video to be inspected, prioritizing the data segments with earlier start times. In practice, multiple sets of data segments to be reviewed are arranged in ascending order of their corresponding start times, and data is extracted starting from the first position in the sorted list until the number of extracted segments reaches the required number of segments to be reviewed. The extracted data segments are then defined as the target subset of segments to be reviewed. This rule is suitable for scenarios where attention is focused on negative content in the early part of the video.

[0096] Specifically, the selection rule based on risk confidence is to sort the data segments to be reviewed according to their degree of negative risk, prioritizing the data segments with the highest risk level. In practice, the negative keyword matching score for each data segment is obtained. Multiple sets of data segments are then arranged in descending order of negative keyword matching score. Starting from the first element in the sorted list, data is extracted until the number of extracted segments reaches the required number of segments to be reviewed. This extracted subset of data segments is then defined as the target set of data segments to be reviewed. This rule is suitable for scenarios with extremely limited computing resources where the most suspicious content must be processed first.

[0097] Specifically, the selection rule based on uniform temporal distribution involves dividing the timeline of the video to be tested into equal intervals. Within each interval, at least one segment of data to be reviewed is selected, ensuring that the selected segments evenly cover the entire video. In practice, the total duration of the video to be tested is divided into multiple time intervals of equal length according to the number of segments to be reviewed. Within each time interval, it is checked whether any segments to be reviewed exist. If they do, the segments within that interval are selected. If multiple segments exist within the same interval, the one with the highest negative keyword match is selected. All segments selected from all intervals are then considered as the target subset of segments to be reviewed. This rule is suitable for scenarios requiring globally uniform detection of the video to avoid missing negative content in specific time segments.

[0098] In practice, multiple sets of data segments to be reviewed are processed according to any of the above-mentioned preset selection rules. When the total number of multiple sets of data segments to be reviewed is less than or equal to the number of segments to be reviewed, no filtering or truncation is required, and all data segments to be reviewed are directly determined as the target subset of segments to be reviewed. When the total number of multiple sets of data segments to be reviewed is greater than the number of segments to be reviewed, they are sorted and truncated according to the preset selection rules, and the target subset of segments to be reviewed corresponding to the number of segments to be reviewed is selected.

[0099] S405. According to the preset frame extraction strategy, perform frame extraction operation on each target segment to be reviewed in the target segment to be reviewed subset to obtain a key frame image subset corresponding to each target segment to be reviewed. The key frame image subset corresponding to each target segment to be reviewed contains at least two key frame images corresponding to each target segment to be reviewed.

[0100] The preset frame extraction strategy refers to the preset frame extraction rules, including different methods for selecting frame extraction time points, used to determine which frames to extract from the video. The preset frame extraction strategy includes a first frame extraction strategy, a second frame extraction strategy, and a third frame extraction strategy.

[0101] The keyframe image subset refers to the set of at least two key image frames extracted after performing frame extraction on a single target segment to be reviewed.

[0102] Specifically, S405 includes the following steps S4101-S4105.

[0103] S4101. Obtain the time range corresponding to each target segment to be reviewed in the target segment to be reviewed subset.

[0104] Specifically, the timestamp attribute carried by each target segment to be reviewed in the target segment to be reviewed subset is read, and the start time point and end time point corresponding to each target segment to be reviewed are extracted from it. The start time point and end time point constitute the time range corresponding to each target segment to be reviewed.

[0105] S4102. When the preset frame extraction strategy is the first frame extraction strategy, the start time point of the video to be detected and the end time point within the time range corresponding to each target segment to be reviewed are determined as the set of frame extraction time points corresponding to each target segment to be reviewed; or... S4103. When the preset frame extraction strategy is the second frame extraction strategy, the start time point, intermediate time point, and end time point of the time range corresponding to each target segment to be reviewed are determined as the set of frame extraction time points corresponding to each target segment to be reviewed, wherein the intermediate time point is the midpoint between the start time point and the end time point; or... S4104. When the preset frame extraction strategy is the third frame extraction strategy, based on a fixed time interval and / or audio volume change and / or audio speech rate change, the frame extraction time point is determined within the time range corresponding to each target segment to be reviewed, and a set of frame extraction time points corresponding to each target segment to be reviewed is obtained. S4105. Based on the set of frame extraction time points corresponding to each target segment to be reviewed, extract the corresponding key image frames from the video to be detected to obtain a key frame image subset corresponding to each target segment to be reviewed.

[0106] The first frame extraction strategy refers to the extraction rule that uses the start time of the video and the end time of each segment as the frame extraction time point.

[0107] The second frame extraction strategy refers to using the start time point, middle time point, and end time point of each segment's time range as the extraction rules for frame extraction time points.

[0108] The intermediate time point refers to the moment that is exactly in the middle between the start time point and the end time point within the time range of the target segment to be reviewed.

[0109] The third frame extraction strategy refers to the extraction rules that determine the frame extraction time point based on a fixed time interval or changes in audio features.

[0110] The set of frame extraction time points refers to the set of all determined frame extraction time points in a single target segment to be reviewed.

[0111] In the specific implementation of S4102, when the preset frame extraction strategy is determined to be the first frame extraction strategy, the start time point (e.g., zero time) of the video to be detected is obtained, and the end time point in the time range corresponding to the current target segment to be reviewed is obtained. The start time point and the end time point of the video to be detected are combined into the set of frame extraction time points corresponding to the target segment to be reviewed.

[0112] In the specific implementation of S4103, when the preset frame extraction strategy is determined to be the second frame extraction strategy, the start time point (T1) and end time point (T2) of the time range corresponding to the current target segment to be reviewed are obtained, the midpoint between the start time point and the end time point (T3=T1+(T2-T1) / 2) is calculated as the midpoint time point, and the three time points of the start time point, midpoint time point and end time point are combined into the set of frame extraction time points corresponding to the target segment to be reviewed.

[0113] In the specific implementation of S4104, when the preset frame extraction strategy is determined to be the third frame extraction strategy, frame extraction time points are set at fixed time intervals (e.g., every five seconds) within the time range corresponding to the target segment to be reviewed; or the volume change curve of the audio data within the time range is extracted, and frame extraction time points are set at the moment when the volume changes abruptly; or the speech rate change curve of the audio data is extracted, and frame extraction time points are set at the moment when the speech rate changes abruptly; one or more frame extraction time points determined by the above methods are combined to obtain the set of frame extraction time points corresponding to the target segment to be reviewed.

[0114] For reference Figure 3 , Figure 3 This diagram illustrates the distribution of frame extraction time points within the time range of the segment to be reviewed, for the three preset frame extraction strategies S4102-S4104. Figure 3The horizontal axis represents the video timeline, and the arrows point in the direction of increasing time. `ts` represents the start time of the time range corresponding to the target segment to be reviewed, and `te` represents the end time of the time range corresponding to the target segment to be reviewed. A blank rectangle represents the time range corresponding to the target segment to be reviewed on the timeline; the left boundary of the blank rectangle corresponds to the start time of the segment, and the right boundary corresponds to the end time of the segment. Solid circles represent fixed-interval frame extraction points, and hollow circles represent audio feature trigger points.

[0115] like Figure 3 As shown in (a), the set of time points for frame extraction corresponding to the first frame extraction strategy consists of two time points: frame extraction point ① corresponds to the start time point of the video to be detected (t=0, i.e., the zero time of the video), and frame extraction point ② corresponds to the end time point te of the current target segment to be reviewed. The above two time points are combined to form the set of time points for frame extraction corresponding to the target segment to be reviewed, i.e., the set of time points for frame extraction = {t=0, te}.

[0116] like Figure 3 As shown in (b), the set of time points for the second frame extraction strategy consists of three time points: frame extraction point ① corresponds to the start time point T1 of the current target segment to be reviewed, frame extraction point ③ corresponds to the end time point T2 of the current target segment to be reviewed, and frame extraction point ② corresponds to the intermediate time point T3 between T1 and T2, where T3 = T1 + (T2 - T1) / 2. The 1 / 2 mark in the figure shows the positional relationship of T3 dividing the segment time range into two equal parts. The above three time points are taken as the set of time points for the target segment to be reviewed, that is, the set of time points for the frame extraction = {T1, T3, T2}.

[0117] like Figure 3 As shown in (c), the set of frame extraction time points corresponding to the third frame extraction strategy is composed of fixed interval time points and / or audio feature trigger time points. Here, ts and te are the start and end time points of the target segment to be reviewed, respectively; Δt represents a fixed time interval (e.g., every 5 seconds); t1 to t6 represent the frame extraction time points set sequentially within the segment time range according to the fixed time interval Δt, where t1 coincides with ts and t6 coincides with te; the two time points indicated by the hollow circles represent audio feature trigger points determined at the moment when the volume or speech rate changes abruptly, based on the volume change curve or speech rate change curve of the audio data. The position of these time points is determined by the actual changes in the audio signal and is offset from the fixed interval time points. Combining the above fixed interval time points and / or audio feature trigger time points yields the set of frame extraction time points corresponding to the target segment to be reviewed, i.e., the set of frame extraction time points = fixed interval time points and / or volume change time points and / or speech rate change time points.

[0118] In the specific implementation of S4105, based on each time point in the set of extracted time points corresponding to each target segment to be reviewed, the location is performed in the video stream of the video to be inspected, the image corresponding to that time point is decoded and extracted, all extracted image images are determined as key image frames, and all key image frames extracted under the same target segment to be reviewed are combined to obtain a key frame image subset corresponding to each target segment to be reviewed.

[0119] As can be seen, in the embodiments of S4101-S4105, by obtaining the time range corresponding to each target segment to be reviewed, and flexibly determining the set of frame extraction time points according to the first frame extraction strategy, the second frame extraction strategy, or the third frame extraction strategy, the system can adopt a matching extraction method for video segments with different characteristics. The second frame extraction strategy ensures comprehensive coverage of the image within the time range of the negative segment by extracting the start time point, the middle time point, and the end time point, avoiding the missed detection caused by extraction at a single time point. The third frame extraction strategy determines the frame extraction time point based on a fixed time interval and changes in audio volume and audio speech rate, realizing targeted extraction of segments with abnormal volume or speech rate, improving the capture accuracy of key frame images, and thus improving the accuracy of subsequent visual review judgment.

[0120] S406. Summarize the keyframe image subsets corresponding to all target segments to be reviewed to obtain the keyframe image set corresponding to the suspected video range.

[0121] Specifically, the keyframe image subsets corresponding to all target segments to be reviewed are merged and stitched together. After removing any possible duplicate image frames, all remaining keyframe images are integrated into a total image set, which is the keyframe image set corresponding to the suspected video range.

[0122] For example, assuming the video to be detected is 30 minutes long, step S402 determines that there are 6 segments to be reviewed; the parsed set of suspected video ranges includes 8 segments; 6 target segments to be reviewed are selected according to preset rules; the second frame extraction strategy is adopted to extract keyframes from the start, middle and end time points of each segment, and further extract frame images of the corresponding time points from the video. Each segment yields 3 keyframe images, and finally a total of 18 keyframe images are collected to form a keyframe image set.

[0123] As can be seen, in the embodiments of S401-S406, the number of segments to be reviewed is determined by obtaining the video duration corresponding to the video to be detected, and a target subset of segments to be reviewed corresponding to the number of segments to be reviewed is selected from multiple sets of segment data based on preset selection rules. This allows the system to limit the scale of segments that actually perform frame extraction operations when there are too many suspected negative segments, avoiding the waste of computing resources caused by redundant frame extraction operations. Subsequently, frame extraction operations are performed only on the target subset of segments to be reviewed and the key frame image set is obtained by summarizing, which reduces the overall data processing volume of the system.

[0124] As can be seen, this embodiment, by combining a preset frame extraction strategy with video duration and the number of segments to be reviewed, can accurately select key time points for frame extraction, ensuring coverage of key scenes suspected of containing negative content. This avoids the resource waste associated with full frame extraction, achieving reasonable allocation and effective utilization of computing resources, and improving the accuracy and processing efficiency of negative information identification.

[0125] S50. Perform visual analysis on the keyframe image set, identify negative content in the keyframe image set, and obtain the target visual verification result corresponding to the keyframe image set.

[0126] Visual analysis refers to the process of parsing and processing image data in a keyframe image set, extracting visual semantic features and embedded text features from the image, and then combining and classifying them with time series data to identify whether the image contains negative content.

[0127] Among them, the target visual verification result refers to the final output of the visual analysis of the keyframe image set, which represents the judgment conclusion on whether the visual dimension contains negative content.

[0128] In one embodiment, the step of performing visual analysis on the keyframe image set to identify negative content in the keyframe image set and obtain a target visual verification result corresponding to the keyframe image set includes: S501. Obtain each keyframe image from the keyframe image set.

[0129] Specifically, the keyframe image set is read, the elements in the set are traversed, and each frame of static image data is read sequentially in chronological order to obtain each keyframe image.

[0130] S502. Perform dual-path visual feature extraction on each keyframe image to obtain single-frame structured visual data corresponding to each keyframe image. The single-frame structured visual data includes single-frame visual semantic features and single-frame text sentiment determination results.

[0131] Among them, the dual-path visual feature extraction operation refers to the parallel processing of extracting visual semantic features and determining the emotional content of text in the keyframe image at the same time, in order to obtain information from different dimensions of the image.

[0132] Among them, single-frame structured visual data refers to the combined data obtained by extracting dual-path visual features from a single keyframe image, which includes single-frame visual semantic features and single-frame text sentiment judgment results.

[0133] Among them, single-frame visual semantic features refer to the feature representations extracted from keyframe images that describe visual content such as objects, scenes, and actions in the picture.

[0134] Among them, the single-frame text sentiment determination result refers to the conclusive label obtained after judging the sentiment tendency and negativeness of the text extracted from the keyframe image.

[0135] Specifically, S502 includes the following steps S5101-S5104.

[0136] S5101. Based on the preset visual language model, extract visual semantic features from each keyframe image to obtain the visual semantic information corresponding to each keyframe image.

[0137] Among them, the pre-trained visual language model refers to a model that is pre-trained to receive image input and output corresponding visual semantic description information.

[0138] Among them, visual semantic information refers to the textual description data output by the preset visual language model, which describes the object categories, scene types and interactive actions contained in the keyframe image.

[0139] In practice, each keyframe image is input into a preset visual language model. The preset visual language model analyzes the pixel distribution in the keyframe image, identifies the object categories, scene types and interactive actions contained in the image, and outputs text data describing the content of the screen. This text data is then determined as the visual semantic information corresponding to each keyframe image.

[0140] S5102. Perform text extraction operation on each keyframe image to obtain the extracted text corresponding to each keyframe image.

[0141] Among them, text extraction refers to the process of identifying and obtaining embedded text content from an image.

[0142] Among them, text extraction from images refers to the text content identified from keyframe images through text extraction operations.

[0143] In practice, each keyframe image is scanned using an optical character recognition algorithm to locate the text region in the image. The pixels of the text region are then subjected to shape recognition and character matching. The recognized strings are combined into text, and this text is identified as the text extracted from the image corresponding to each keyframe image.

[0144] S5103 performs sentiment analysis and negative judgment on the text extracted from the image corresponding to each key frame, and obtains the sentiment judgment result of the text corresponding to each key frame.

[0145] Specifically, the extracted text corresponding to each keyframe image is input into the sentiment classification engine. The semantic tendency of the extracted text is calculated to determine the probability that the extracted text is positive, neutral or negative, and whether the extracted text contains preset negative words. Based on the combination of semantic tendency and negative word matching, the sentiment judgment result of the text corresponding to each keyframe image is output.

[0146] S5104 performs a structured mapping between the visual semantic information corresponding to each keyframe image and the text sentiment determination result corresponding to each keyframe image, to obtain single-frame structured visual data corresponding to each keyframe image.

[0147] Specifically, visual semantic information is transformed into single-frame visual semantic feature vectors, and text sentiment determination results are mapped into single-frame text sentiment feature vectors. The single-frame visual semantic feature vectors and single-frame text sentiment feature vectors are aligned and spliced ​​according to a preset data structure to generate a structured data volume. This structured data volume is determined as the single-frame structured visual data corresponding to each keyframe image. This single-frame structured visual data contains single-frame visual semantic features and single-frame text sentiment determination results.

[0148] As can be seen, in the embodiments of S5101-S5104, image content features and embedded text features can be extracted in parallel, overcoming the limitation that a single visual feature cannot identify embedded negative text in an image, and realizing the structured alignment of risk information in the visual dimension and the text dimension, providing a data foundation for subsequent feature fusion.

[0149] S503. Perform feature fusion on the single-frame visual semantic features and single-frame text emotion judgment results corresponding to each keyframe image to obtain the single-frame fused features corresponding to each keyframe image.

[0150] Feature fusion refers to the process of combining and mapping feature data of different modalities or different dimensions to generate a unified feature representation.

[0151] Among them, single-frame fusion features refer to the comprehensive feature vector obtained by fusing single-frame visual semantic features with single-frame text sentiment judgment results.

[0152] In practice, the single-frame visual semantic features and single-frame text sentiment judgment results corresponding to each keyframe image are input into the feature fusion layer. The modal differences between the two features are eliminated through matrix transformation and multiplication operations. The two features are mapped to a unified feature space and merged into a feature vector. This feature vector is determined as the single-frame fusion feature corresponding to each keyframe image.

[0153] S504. Based on the temporal order of each frame in the keyframe image set, extract the temporal features of the single-frame fusion features corresponding to each frame to obtain the global temporal features corresponding to the keyframe image set.

[0154] Temporal feature extraction refers to the process of calculating the contextual dependencies of feature sequences according to their chronological order.

[0155] Among them, global temporal features refer to the feature representation that represents the temporal dimension change pattern of the entire suspected negative video segment, obtained by extracting temporal features from the single-frame fusion features corresponding to each frame in the keyframe image set.

[0156] In practice, according to the timestamp information of each frame in the keyframe image set, the single-frame fusion features of each frame are arranged into a feature sequence. This feature sequence is then input into the temporal feature extraction network. The temporal feature extraction network calculates the feature association between adjacent frames and interval frames, captures the dynamic change pattern of the image content over time, and outputs a feature representation that integrates time dimension information. This feature representation is then determined as the global temporal feature corresponding to the keyframe image set.

[0157] S505. Map and classify the global temporal features to obtain the target visual verification result.

[0158] Mapping classification refers to the process of inputting feature data into a classifier to calculate the probability of it belonging to each preset category and outputting the corresponding classification result.

[0159] In practice, the global temporal features are input into the fully connected classification layer. The fully connected classification layer calculates the first probability value of the global temporal features belonging to the positive content category, the second probability value belonging to the neutral content category, and the third probability value belonging to the negative content category. The category with the highest probability value is selected as the judgment conclusion of the visual review, and this judgment conclusion is determined as the target visual review result.

[0160] As can be seen, in the embodiments of S501-S505, not only can the negative risks of a single frame image be determined by combining visual and textual dual-modal information, but also the dynamic negative content can be identified by combining the contextual dependencies between previous and subsequent frames. This avoids the misjudgment caused by isolated analysis of a single frame image and improves the accuracy of visual review and determination.

[0161] S60. Based on a preset fusion strategy, the audio recognition result and the target visual verification result are fused and judged to obtain a target judgment result for the video to be detected. The target judgment result is used to indicate whether the video to be detected is a video containing negative information.

[0162] Among them, the preset fusion strategy refers to the preset rules for combining the judgment results of visual modality and text modality and outputting the final conclusion.

[0163] Among them, the target judgment result refers to the conclusion data output after fusing the audio recognition result and the target visual verification result through a preset fusion strategy, which is used to indicate whether the video to be detected is a video containing negative information.

[0164] In one embodiment, the audio recognition result and the target visual verification result are fused and judged based on a preset fusion strategy to obtain a target judgment result for the video to be detected. The target judgment result is used to indicate whether the video to be detected contains negative information, including: S601. Based on the target visual verification results, obtain the visual modality negative confidence level under the visual modality.

[0165] In this context, visual modality refers to the data dimension that conveys information through the content of an image. Therefore, visual modality negative confidence score is a numerical value calculated based on the target visual verification results, representing the probability that negative content exists in the visual image.

[0166] In practice, the visual verification results of the target are analyzed, and the probability value of the target visual verification results belonging to the negative category is extracted. This probability value is determined as the visual modality negative confidence level under the visual modality.

[0167] S602. Based on the audio recognition results, obtain the negative confidence level of the text modality in the text modality.

[0168] Among them, text modality refers to the data dimension that conveys information through text and voice content.

[0169] Among them, the negative confidence score of the text modality refers to the probability value of the existence of negative content in the audio text, which is calculated based on the audio recognition results.

[0170] In a specific implementation, the audio recognition result is obtained, which includes a text sequence with a time alignment relationship; the text sequence is input into a pre-trained text sentiment classification model, and the text sentiment classification model is used to extract and map semantic features of the text sequence, and output the negative probability prediction value of the text sequence at each time step; the negative probability prediction value at each time step is determined as the negative confidence level of the text modality under the text modality.

[0171] S603. If the negative confidence level of the visual modality is greater than or equal to the first preset trigger threshold, or if the negative confidence level of the text modality is greater than or equal to the first preset trigger threshold, then the target judgment result is determined to confirm that the video to be detected is a video containing negative information.

[0172] The first preset trigger threshold refers to a pre-set critical value used to determine whether the single-modal negative confidence level is sufficient to directly confirm that the video contains negative information.

[0173] In practice, the negative confidence scores of the visual modality and the text modality are compared with a first preset trigger threshold (e.g., 0.8). If the negative confidence score of the visual modality is greater than or equal to 0.8, or the negative confidence score of the text modality is greater than or equal to 0.8, it indicates that a single modality has provided sufficient risk evidence. In this case, the system directly determines that the target judgment result is that the video to be detected is a video containing negative information.

[0174] S604, or, if both the visual modal negative confidence score and the text modal negative confidence score are less than the first preset trigger threshold, then multimodal evidence enhancement calculation is performed based on the visual modal negative confidence score and the text modal negative confidence score to obtain the target negative confidence score.

[0175] Among them, multimodal evidence enhancement computation refers to the computational process of combining the confidence levels of multiple modalities to mutually verify and amplify the values ​​when the confidence level of a single modality is insufficient.

[0176] Among them, the target negative confidence score refers to the final negative probability value calculated by multimodal evidence enhancement, which integrates visual and textual modal information.

[0177] In practice, if both the visual modality negative confidence score and the text modality negative confidence score are less than the first preset trigger threshold, it means that neither modality is sufficient to be judged as negative on its own. The negative content recognition system performs multimodal evidence enhancement calculation based on the visual modality negative confidence score and the text modality negative confidence score to obtain the target negative confidence score.

[0178] S605. If the target negative confidence level is greater than or equal to the preset fusion judgment threshold, then the target judgment result is determined to confirm that the video to be detected is a video containing negative information.

[0179] Among them, the preset fusion judgment threshold refers to the pre-set critical value used to determine whether the negative confidence level of the target reaches the standard for confirming that the video contains negative information.

[0180] In practice, the negative confidence level of the target is compared with the preset fusion judgment threshold (e.g., 0.75). If the negative confidence level of the target is greater than or equal to 0.75, it means that the multimodal joint evidence has reached the risk standard, and the system determines that the target judgment result is that the video to be detected is a video containing negative information.

[0181] S606. If the target negative confidence level is less than the preset fusion judgment threshold, then the target judgment result is determined to be that the video to be detected is a video that does not contain negative information.

[0182] In practice, if the negative confidence level of the target is less than the preset fusion judgment threshold, it means that the multimodal joint evidence has not yet reached the risk standard, and the negative content recognition system determines that the target judgment result is that the video to be detected does not contain negative information.

[0183] As can be seen, in this embodiment, the visual modality negative confidence and text modality negative confidence are obtained based on the target visual verification result and the audio recognition result, respectively. When the confidence of either modality is greater than or equal to the first preset trigger threshold, the video to be detected is directly confirmed as containing negative information, realizing the rapid determination of strong evidence in a single modality and avoiding unnecessary fusion computing resource consumption. When both modal confidences are less than the first preset trigger threshold, the target negative confidence is obtained through multimodal evidence enhancement calculation and combined with the preset fusion judgment threshold for final judgment. This enables the system to effectively identify obscure negative content through multimodal joint analysis when single modal evidence is insufficient, avoiding the missed detection phenomenon caused by single modality judgment and improving recall and accuracy.

[0184] In a specific implementation of S604, optionally, the step of performing multimodal evidence enhancement calculation based on the visual modality negative confidence and the text modality negative confidence to obtain the target negative confidence includes: comparing the modal confidence of the visual modality negative confidence and the text modality negative confidence to determine the primary modality and the secondary modality; generating an enhancement factor based on the confidence corresponding to the secondary modality; using the enhancement factor to perform feature enhancement on the confidence of the primary modality to obtain the enhanced modality confidence; and fusing and mapping the enhanced modality confidence with the confidence of the secondary modality to obtain the target negative confidence.

[0185] Modal confidence comparison refers to the operation of comparing the numerical values ​​of the visual modal negative confidence score with the text modal negative confidence score.

[0186] Among them, the dominant modality refers to the modality with a larger negative confidence score in the visual modality and the negative confidence score in the text modality, representing a data dimension with stronger evidence.

[0187] Among them, the auxiliary modality refers to the modality with a smaller negative confidence score in the visual modality and the negative confidence score in the text modality, and is used to provide data dimensions for corroborating evidence.

[0188] The enhancement factor is a coefficient calculated based on the confidence level of the auxiliary mode, used to numerically amplify the confidence level of the main mode.

[0189] Among them, enhanced modal confidence refers to the new confidence value obtained after feature enhancement of the confidence of the main modality using enhancement factors.

[0190] Among them, fusion mapping refers to the operation of merging the confidence scores of the enhanced mode and the auxiliary mode and outputting the final value.

[0191] In practice, the negative confidence scores of the visual modality and the text modality are compared. The modality with the larger score is identified as the primary modality, and the modality with the smaller score is identified as the secondary modality. The confidence score corresponding to the secondary modality is obtained by inputting the confidence score of the secondary modality into an exponential or logarithmic function for calculation, generating a value greater than 1, which is determined as the enhancement factor. The higher the confidence score of the secondary modality, the larger the generated enhancement factor. The enhancement factor is multiplied by the confidence score of the primary modality, and the result of the product is determined as the enhancement modality confidence score. The enhancement modality confidence score and the secondary modality confidence score are summed or weighted averaged, and the calculation result is mapped to a numerical range of 0 to 1. This mapped value is determined as the target negative confidence score.

[0192] As can be seen, this embodiment can use the corroborating information of the auxiliary modality to numerically amplify the confidence of the main modality, highlighting the dominant role of the main modality in the judgment process. At the same time, the participation of the auxiliary modality enhances the robustness of the judgment result and improves the recognition sensitivity of the system when the evidence of a single modality is weak but the multimodal directions are consistent.

[0193] In a specific implementation of S604, optionally, the step of performing multimodal evidence enhancement calculation based on the visual modal negative confidence and the text modal negative confidence to obtain the target negative confidence includes: performing multimodal interaction on the visual modal negative confidence and the text modal negative confidence to obtain cross-modal association features; determining a first dynamic weight corresponding to the visual modality and a second dynamic weight corresponding to the text modality based on the cross-modal association features; and performing weighted fusion of the visual modal negative confidence and the text modal negative confidence based on the first dynamic weight and the second dynamic weight to obtain the target negative confidence.

[0194] Multimodal interaction refers to the process of cross-calculating visual modal features and text modal features to extract the correlation between the two modalities.

[0195] Among them, cross-modal association features refer to the feature representations extracted through multimodal interaction that indicate the mutual corroboration relationship between visual content and audio text content.

[0196] The first dynamic weight refers to the adaptive coefficient calculated based on cross-modal association features and assigned to the negative confidence score of the visual modality.

[0197] The second dynamic weight refers to the adaptive coefficients calculated based on cross-modal association features and assigned to the negative confidence of the text modality.

[0198] Weighted fusion refers to the data processing operation of multiplying and summing the confidence scores according to the corresponding dynamic weights.

[0199] In practice, cross-attention calculation is performed on the feature vectors corresponding to the negative confidence scores of the visual modality and the text modality to extract the mutual corroboration relationship between the visual content and the audio text content. The extracted relationship features are identified as cross-modal association features. The cross-modal association features are input into a fully connected network layer to calculate the importance coefficients of the visual modality and the text modality under the current association relationship. The two importance coefficients are converted into probability values ​​that sum to 1 using a normalization function. The probability value corresponding to the visual modality is identified as the first dynamic weight, and the probability value corresponding to the text modality is identified as the second dynamic weight. The first dynamic weight is multiplied by the first product of the negative confidence score of the visual modality, and the second dynamic weight is multiplied by the negative confidence score of the text modality. The first product and the second product are added together, and the sum is identified as the target negative confidence score.

[0200] As can be seen, this embodiment can adaptively adjust the weight ratio of each modality based on cross-modal association features, dynamically allocate higher confidence modalities with larger weight coefficients for different scenarios, avoid the imbalance of modal contributions caused by fixed weights, and improve the accuracy of multimodal fusion judgment results.

[0201] In a specific implementation of S604, optionally, the step of performing multimodal evidence enhancement calculation based on the visual modality negative confidence and the text modality negative confidence to obtain the target negative confidence includes: performing discrete wavelet transform on the visual modality negative confidence to obtain low-frequency subband components and high-frequency subband components of visual confidence; performing discrete wavelet transform on the text modality negative confidence to obtain low-frequency subband components and high-frequency subband components of text confidence; performing global trend fusion on the low-frequency subband components of visual confidence and the low-frequency subband components of text confidence to obtain a fused low-frequency component; performing detail complementarity enhancement on the high-frequency subband components of visual confidence and the high-frequency subband components of text confidence to obtain a fused high-frequency component; and performing inverse discrete wavelet transform on the fused low-frequency component and the fused high-frequency component to obtain the target negative confidence.

[0202] Discrete wavelet transform (DWT) is a mathematical transformation method that decomposes a time-domain signal into sub-bands of different frequencies. Unlike Fourier transform, DWT has time-frequency localization properties, enabling it to simultaneously capture the local features of a signal in the time dimension and the scale features in the frequency dimension.

[0203] In this scheme, discrete wavelet transform is applied to the confidence score numerical sequence. By selecting a preset wavelet basis function (such as Haar wavelet, Daubechies wavelet, or Symlet wavelet), the confidence score sequence is decomposed into one or more levels, and the global trend component and local detail component contained in the confidence score sequence are explicitly separated in the frequency domain space.

[0204] The low-frequency sub-band component of visual confidence refers to the low-frequency approximation coefficient sequence obtained after performing a discrete wavelet transform on the visual modality negative confidence sequence, representing the global trend information of visual modality confidence in the time dimension. This global trend information reflects the overall trend of visual negative content over a relatively long time span, corresponding to the basic components in the confidence sequence that change gradually and with large amplitude. For example, if the visual modality negative confidence sequence is [0.82, 0.85, 0.79, 0.88, 0.91, 0.87, 0.83, 0.89], then the low-frequency sub-band component of visual confidence obtained after discrete wavelet transform is approximately [0.836, 0.835, 0.870, 0.860], reflecting a stable trend of overall high negative confidence.

[0205] The high-frequency sub-band component of visual confidence refers to the high-frequency detail coefficient sequence obtained after performing discrete wavelet transform on the visual modality negative confidence sequence, representing the detail fluctuation information of visual modality confidence in the time dimension. This detail fluctuation information reflects the rapid changes in visual negative content within a short time span, corresponding to the drastic but relatively small fluctuation components in the confidence sequence. For example, corresponding to the aforementioned visual confidence sequence, its high-frequency sub-band component might be [-0.021, 0.064, -0.028, 0.042], reflecting the local jitter characteristics of confidence in different time periods.

[0206] The low-frequency sub-band component of text confidence refers to the low-frequency approximation coefficient sequence obtained after performing discrete wavelet transform on the negative confidence sequence of the text modality, representing the global trend information of text modality confidence in the time dimension. The global trend information reflects the overall distribution characteristics of the negative degree of text semantics over a long time span, and together with the low-frequency sub-band component of visual confidence, constitutes the negative evidence of the two modalities at the macro level.

[0207] The high-frequency subband component of text confidence refers to the high-frequency detail coefficient sequence obtained after performing discrete wavelet transform on the negative confidence sequence of the text modality, representing the detail fluctuation information of the text modality confidence in the time dimension. The detail fluctuation information reflects the abrupt change characteristics of the negative degree of text semantics in a short period of time, such as the phenomenon of a sudden increase in confidence caused by the appearance of extremely negative words at a certain moment.

[0208] Global trend fusion refers to the weighted average merging of low-frequency sub-band components from the visual and textual modalities. This weighted average merging aims to integrate consensus-based negative evidence from both modalities at the macro-trend level, ensuring that the fused low-frequency components retain both the overall negative trend of the visual modality and incorporate global semantic negative information from the textual modality, thus forming a globally confident trend representation with enhanced cross-modal consistency.

[0209] Among them, detail complementarity enhancement refers to a fusion operation that differentiates the high-frequency sub-band components of visual confidence and text confidence based on their respective energy intensities. Specifically, the energy values ​​of the high-frequency sub-band components of visual confidence and text confidence are calculated separately. The high-frequency component with stronger energy is identified as the main detail component, and the high-frequency component with weaker energy is multiplied by a preset compensation coefficient and then superimposed on the main detail component, thereby achieving complementary enhancement of detail information and obtaining the fused high-frequency component.

[0210] The inverse discrete wavelet transform (IDT) is the inverse operation of the discrete wavelet transform, a mathematical transformation method that reconstructs the processed low-frequency and high-frequency subband components into a time-domain signal sequence. In this scheme, the IDT reconstructs the target negative confidence score numerical sequence by fusing the low-frequency and high-frequency components. The reconstruction process ensures the consistency of the output result with the original confidence score sequence in terms of numerical dimensions and time dimension.

[0211] In specific implementation, a visual modality negative confidence sequence output by the visual feature extraction module is obtained. This sequence contains negative confidence values ​​corresponding to several time steps, each value ranging from 0 to 1. A preset wavelet basis function is selected, preferably the Daubechies-4 wavelet, to perform a first-level discrete wavelet decomposition on the visual modality negative confidence sequence. The decomposition process consists of two parallel filtering branches: a low-pass filtering branch convolves the visual modality negative confidence sequence with the low-pass filter, the convolution result reflecting the low-frequency smoothing components in the sequence; subsequently, a downsampling operation with a stride of 2 is performed on the convolution result, i.e., every other value is retained, resulting in a low-frequency sub-band component of visual confidence with a length half that of the original sequence. Each value in the low-frequency sub-band component comprehensively reflects the average trend of visual confidence within the corresponding time window, representing the overall intensity distribution of negative content in the visual image over a longer time scale. The high-pass filtering branch convolves the visual modality negative confidence sequence with the high-pass filter. The convolution result reflects the high-frequency fluctuation components in the sequence. Subsequently, a downsampling operation with a stride of 2 is performed to obtain a high-frequency sub-band component of visual confidence with a length half that of the original sequence. Each value in the high-frequency sub-band component of visual confidence reflects the local change amplitude of visual confidence within the corresponding time window, reflecting the degree of abrupt change in negative content of the visual image between adjacent time steps.

[0212] For example, suppose the visual modality negative confidence sequence is 0.82, 0.85, 0.79, 0.88, 0.91, 0.87, 0.83, and 0.89, for a total of 8 time steps. After Haar wavelet decomposition, the low-pass filter branch averages two adjacent values ​​and retains the result, obtaining the low-frequency sub-band components of visual confidence as 0.835, 0.835, 0.890, and 0.860, a total of 4 values, reflecting the overall macro trend of the visual image maintaining a relatively high negative confidence (average of approximately 0.855); the high-pass filter branch halves the difference between two adjacent values ​​and retains the result, obtaining the high-frequency sub-band components of visual confidence as -0.015, -0.045, 0.020, and -0.030, a total of 4 values, reflecting the small fluctuations in the built-in confidence at each time step.

[0213] In practical implementation, if the negative confidence sequence lengths output by the visual modality and the text modality are inconsistent, firstly, zero-padding or linear interpolation alignment is performed on the shorter sequence to ensure that the two modality sequences have the same number of time steps N. Then, the text modality negative confidence sequence output by the text feature extraction module is obtained. This text modality negative confidence sequence is aligned with the visual modality negative confidence sequence in the time dimension, has the same number of time steps, and each value is between 0 and 1. Using the same wavelet basis function and decomposition level as before, discrete wavelet decomposition is performed on the text modality negative confidence sequence. This is processed through low-pass and high-pass filtering branches to obtain the low-frequency and high-frequency sub-band components of the text confidence. The generation process of these two sub-band components is completely consistent with the processing method of the visual modality in the previous step, and will not be repeated here. The low-frequency sub-band component of the text confidence represents the overall trend of the text semantic negativity within each time window; the high-frequency sub-band component of the text confidence represents the local variation details of the text semantic negativity within each time window.

[0214] For example, suppose the negative confidence sequence of the text modality is 0.76, 0.91, 0.88, 0.72, 0.85, 0.93, 0.78, 0.86, for a total of 8 time steps. After Haar wavelet decomposition, the low-frequency sub-band components of the text confidence are obtained as 0.835, 0.800, 0.890, and 0.820, and the high-frequency sub-band components are -0.075, 0.080, -0.040, and -0.040. Comparing the low-frequency sub-band components of the visual and textual modalities, it can be found that both remain in the range of 0.80 to 0.89, indicating that the visual images and audio text have a strong consistency in negative content at the macro level, which is consistent with the characteristic of mutual corroboration of audiovisual content in real negative videos. Comparing the high-frequency subband components reveals that the absolute value of the text high-frequency subband component reaches a maximum of 0.080, which is greater than the maximum absolute value of the visual high-frequency subband component (0.045). This indicates that the text modality fluctuates more dramatically at the detail level and carries richer local discriminative information.

[0215] In specific implementation, an element-wise weighted average operation is performed on the visual confidence low-frequency sub-band components and the text confidence low-frequency sub-band components to obtain the fused low-frequency components. The element-wise weighted average operation is performed as follows: for each pair of values ​​at the same position in the low-frequency sub-band component sequence, the corresponding low-frequency fusion weight is multiplied and then summed to obtain the value of the fused low-frequency component at that position. The sum of the visual low-frequency fusion weight and the text low-frequency fusion weight is 1 to ensure that the numerical range of the fusion result is consistent with that of the original low-frequency sub-band components. The visual low-frequency fusion weight and the text low-frequency fusion weight are determined as follows: the arithmetic mean of all values ​​in the visual confidence low-frequency sub-band component sequence and the arithmetic mean of all values ​​in the text confidence low-frequency sub-band component sequence are calculated respectively; the visual low-frequency sub-band component mean is divided by the sum of the means of the two modal low-frequency sub-band components to obtain the visual low-frequency fusion weight; the text low-frequency sub-band component mean is divided by the sum of the means of the two modal low-frequency sub-band components to obtain the text low-frequency fusion weight. The weighting method allows modalities with richer global trend information (i.e., higher mean and stronger overall negative tendency) to occupy a larger proportion in the fusion of low-frequency components, thereby achieving adaptive trend integration based on content credibility.

[0216] For example, continuing with the numerical examples above, the visual confidence low-frequency sub-band components are 0.835, 0.835, 0.890, and 0.860, with an arithmetic mean of 0.855; the text confidence low-frequency sub-band components are 0.835, 0.800, 0.890, and 0.820, with an arithmetic mean of 0.836. Dividing the visual low-frequency mean of 0.855 by the sum of the two (1.691), we get a visual low-frequency fusion weight of approximately 0.506; dividing the text low-frequency mean of 0.836 by the sum of the two (1.691), we get a text low-frequency fusion weight of approximately 0.494. Applying these weights element-wise to the low-frequency sub-band components of both modalities, we obtain fused low-frequency components of 0.835, 0.818, 0.890, and 0.840. The fused low-frequency component comprehensively reflects the trend information of both visual and textual modalities pointing to high negative confidence at the macro level, and has stronger cross-modal evidence support compared with the low-frequency sub-band component of a single modality.

[0217] In specific implementation, signal energy values ​​are calculated for both the visual confidence high-frequency sub-band component and the text confidence high-frequency sub-band component. The signal energy value is calculated by squaring each value in the high-frequency sub-band component sequence, then summing all the squared values ​​to obtain the total energy value of that high-frequency sub-band component. A larger energy value indicates richer detail fluctuation information carried by the high-frequency sub-band component of that mode, and a stronger local discrimination capability.

[0218] Based on the comparison between the energy values ​​of the visual confidence high-frequency subband components and the text confidence high-frequency subband components, the principal detail component and the compensated detail component are determined: if the energy value of the visual confidence high-frequency subband component is greater than or equal to the energy value of the text confidence high-frequency subband component, then the visual confidence high-frequency subband component is determined as the principal detail component, and the text confidence high-frequency subband component is determined as the compensated detail component; if the energy value of the visual confidence high-frequency subband component is less than the energy value of the text confidence high-frequency subband component, then the text confidence high-frequency subband component is determined as the principal detail component, and the visual confidence high-frequency subband component is determined as the compensated detail component.

[0219] After determining the primary detail components and compensated detail components, the dynamic compensation coefficient is calculated. The dynamic compensation coefficient is calculated by dividing the smaller of the energy values ​​of the two high-frequency sub-band components by the sum of the larger value and a preset smoothing constant. The physical meaning of the dynamic compensation coefficient is as follows: the smaller the difference in high-frequency energy between the two modes, the closer the compensation coefficient is to 1, and the stronger the contribution of the compensated detail components to the primary detail components; the larger the difference in high-frequency energy between the two modes, the closer the compensation coefficient is to 0, avoiding noise interference from low-discrimination detail information to the primary detail components. After obtaining the dynamic compensation coefficient, each value in the compensated detail component sequence is multiplied by the dynamic compensation coefficient, and the product is added element-wise to the value corresponding to the position in the primary detail component sequence to obtain the fused high-frequency component.

[0220] For example, continuing with the numerical examples above, the high-frequency sub-band components of visual confidence are -0.015, -0.045, 0.020, and -0.030. Squaring each value and summing them yields a visual high-frequency energy value of 0.003550. The high-frequency sub-band components of text confidence are -0.075, 0.080, -0.040, and -0.040. Squaring each value and summing them yields a text high-frequency energy value of 0.015225. Since the text high-frequency energy value of 0.015225 is greater than the visual high-frequency energy value of 0.003550, the text confidence high-frequency sub-band component is determined to be the principal detail component, and the visual confidence high-frequency sub-band component is determined to be the compensating detail component. The dynamic compensation coefficient is calculated by dividing the smaller value of 0.003550 by the larger value of 0.015225, resulting in a compensation coefficient of approximately 0.233. Each value in the visual high-frequency sub-band component sequence is multiplied by 0.233 and then added to the corresponding value in the text high-frequency sub-band component sequence to obtain fused high-frequency components of -0.0785, 0.0695, -0.0353, and -0.0470. These fused high-frequency components are primarily based on detail fluctuations in the text modality while incorporating detail compensation from the visual modality. Compared to using high-frequency sub-band components from either modality alone, they contain more complete local discrimination details.

[0221] In specific implementation, the fused low-frequency component and the fused high-frequency component are used as inputs to the inverse discrete wavelet transform (IDE). The same wavelet basis functions as in the previous step are used to perform the IDE operation. The execution process of the IDE is as follows: First, interpolation upsampling is performed on the fused low-frequency component and the fused high-frequency component, i.e., 0 is inserted between every two adjacent values, restoring the length of the two sequences from half N to N. Then, the upsampled low-frequency and high-frequency component sequences are convolved with their corresponding reconstruction filters, with the low-frequency component sequence convolved with a low-pass reconstruction filter and the high-frequency component sequence convolved with a high-pass reconstruction filter. Finally, the two convolution results are added element-wise at corresponding positions to obtain the reconstructed target negative confidence sequence. The reconstruction result is then truncated, with values ​​exceeding the 0-1 range truncated to their corresponding boundary values ​​to ensure that each element of the target negative confidence sequence satisfies the valid range constraint of the probability value. The final sequence is then determined as the target negative confidence.

[0222] For example, the fused low-frequency components 0.835, 0.818, 0.890, and 0.840 are reconstructed using the fused high-frequency components -0.0785, 0.0695, -0.0353, and -0.0470 via Haar wavelet inverse transform. The inverse transform process performs addition and subtraction operations on each pair of corresponding values ​​between the low-frequency and high-frequency components: subtracting the first pair of values ​​(0.835 and -0.0785) yields 0.757, and adding them together yields 0.914; the second pair of values ​​(0.818)... Subtracting 0.0695 from the first value gives 0.749, and adding the first value gives 0.888. Subtracting the third logarithm 0.890 from -0.0353 gives 0.855, and adding the second value gives 0.925. Subtracting the fourth logarithm 0.840 from -0.0470 gives 0.793, and adding the third value gives 0.887. The final target negative confidence sequence is 0.757, 0.914, 0.749, 0.888, 0.855, 0.925, 0.793, and 0.887. The mean of the target negative confidence sequence is 0.846, which is highly consistent with the original visual modality mean of 0.854 and the text modality mean of 0.836. While preserving the global trend of consensus between the two modalities, it effectively enhances the local detail discrimination information.

[0223] As can be seen, this scheme introduces discrete wavelet transform to decompose the confidence scores of visual and textual modalities from the time domain to the frequency domain, achieving fine-grained and interpretable cross-modal fusion. On the one hand, it adaptively weights and fuses the low-frequency components that characterize macro trends, strengthening the negative consensus evidence across modalities. On the other hand, it dynamically complements and enhances the high-frequency components that characterize local mutations based on signal energy, effectively capturing and integrating the instantaneous negative burst features of a single modality, avoiding the smoothing of detailed information.

[0224] This embodiment first performs text conversion and emotion / negative content recognition on the audio. Only the audio recognition results carrying the tag requiring frame extraction for verification are used to determine the range of suspected videos. Frame extraction is then performed only within this range using a preset frame extraction strategy for subsequent visual analysis. This avoids full frame extraction and full visual analysis of the entire video to be detected, thereby reducing computational resource consumption and improving the overall system processing efficiency. Simultaneously, negative content recognition is performed on the keyframe image set within the suspected range through visual analysis to obtain the target visual verification result. Then, based on a preset fusion strategy, the audio recognition result and the visual verification result are fused for judgment, achieving joint decision-making using audio and visual dual-modal information. This effectively overcomes the limitations of single-modal analysis and improves the accuracy and robustness of negative content recognition.

[0225] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0226] As another aspect of the embodiments of this application, this application provides a negative information identification device. The negative information identification device can be a software module, which includes several instructions stored in a memory. A processor can access the memory and execute the instructions to complete the negative information identification method described in the various embodiments above.

[0227] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a negative information recognition device provided in an embodiment of this application. Figure 4 As shown, the negative information recognition device 400 includes: Processing unit 401 is used to convert the audio data in the acquired video to be detected into text to obtain the target text; The processing unit 401 is also used to perform sentiment analysis and negative recognition processing on the target text to obtain an audio recognition result carrying a frame-extraction verification label; The processing unit 401 is further configured to determine a set of suspected video ranges in the video to be detected based on the audio recognition results carrying the frame-sampling verification tag. The processing unit 401 is further configured to perform frame extraction operation on the suspected video range set using a preset frame extraction strategy to obtain a set of key frame images corresponding to the suspected video range. Analysis unit 402 is used to perform visual analysis on the keyframe image set, identify negative content in the keyframe image set, and obtain target visual verification results corresponding to the keyframe image set; The analysis unit 402 is further configured to perform a fusion judgment on the audio recognition result and the target visual verification result based on a preset fusion strategy to obtain a target judgment result for the video to be detected. The target judgment result is used to indicate whether the video to be detected is a video containing negative information.

[0228] This implementation first performs text conversion and emotion / negative content recognition on the audio. Only audio recognition results carrying tags requiring frame extraction for verification are used to determine the range of suspected videos. Frame extraction is then performed only within this suspected video range using a preset strategy for subsequent visual analysis. This avoids full frame extraction and full visual analysis of the entire video being tested, thus reducing computational resource consumption and improving overall system efficiency. Simultaneously, negative content recognition is performed on keyframe images within the suspected range through visual analysis to obtain target visual verification results. Then, based on a preset fusion strategy, the audio recognition results and visual verification results are fused for judgment, achieving joint decision-making using dual-modal audio and visual information. This effectively overcomes the limitations of single-modal analysis and improves the accuracy and robustness of negative content recognition.

[0229] It should be noted that the aforementioned negative information identification device can execute the negative information identification method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the negative information identification device can be found in the negative information identification method provided in the embodiments of this application.

[0230] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by the processor, cause the processor to perform the negative information identification method as described in the foregoing embodiments.

[0231] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0232] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for identifying negative information, characterized in that, The method includes: The audio data in the acquired video to be detected is converted into text to obtain the target text; Sentiment analysis and negative recognition processing are performed on the target text to obtain audio recognition results carrying tags that need to be extracted for verification; Based on the audio recognition results carrying the tags that need to be reviewed by frame extraction, a set of suspected video ranges is determined in the video to be detected; A preset frame extraction strategy is used to perform frame extraction on the suspected video range set to obtain a set of keyframe images corresponding to the suspected video range, including: Obtain a subset of target segments to be reviewed from the set of keyframe images, and obtain the time range corresponding to each target segment to be reviewed in the subset of target segments to be reviewed; when the preset frame extraction strategy is the third frame extraction strategy, set frame extraction time points at fixed time intervals within the time range corresponding to each target segment to be reviewed, and set audio feature trigger time points at abrupt changes based on changes in audio volume and / or audio speech rate. The audio feature trigger time points and the frame extraction time points set at the fixed time intervals are staggered in position. Combine the frame extraction time points set at the fixed time intervals and the audio feature trigger time points into a set of frame extraction time points corresponding to each target segment to be reviewed; according to the set of frame extraction time points corresponding to each target segment to be reviewed, extract the corresponding key image frames from the video to be detected to obtain a subset of keyframe images corresponding to each target segment to be reviewed. Visual analysis is performed on the keyframe image set to identify negative content in the keyframe image set and obtain the target visual verification result corresponding to the keyframe image set. The audio recognition result and the target visual verification result are fused and judged based on a preset fusion strategy to obtain a target judgment result for the video to be detected. The target judgment result is used to indicate whether the video to be detected contains negative information, including: If both the visual modal negative confidence score and the text modal negative confidence score are less than a first preset trigger threshold, then multimodal evidence enhancement calculation is performed based on the visual modal negative confidence score and the text modal negative confidence score to obtain the target negative confidence score, including: The visual modality negative confidence score is subjected to discrete wavelet transform to obtain low-frequency and high-frequency sub-band components of visual confidence score; the text modality negative confidence score is subjected to discrete wavelet transform to obtain low-frequency and high-frequency sub-band components of text confidence score; global trend fusion is performed on the low-frequency and text confidence sub-band components of visual confidence score to obtain fused low-frequency components; detail complementarity enhancement is performed on the high-frequency and text confidence sub-band components of visual confidence score to obtain fused high-frequency components; and inverse discrete wavelet transform is performed on the fused low-frequency and fused high-frequency components to obtain the target negative confidence score.

2. The method according to claim 1, characterized in that, The process of performing sentiment analysis and negative recognition on the target text to obtain audio recognition results carrying tags that require frame extraction and verification includes: Sentiment analysis is performed on the target text to obtain the sentiment polarity result of the target text; Using multiple subtexts in the target text as query identifiers, negative keyword matching is performed in a preset negative keyword database, and the negative keyword matching degree corresponding to each subtext is calculated based on the character overlap. Based on the negative keyword matching degree corresponding to each subtext and the sentiment polarity result of the target text, an audio recognition result carrying a label that needs to be extracted for verification is obtained.

3. The method according to claim 2, characterized in that, The step of obtaining audio recognition results carrying tags requiring frame extraction and verification based on the negative keyword matching degree corresponding to each sub-text and the sentiment polarity result of the target text includes: When the negative keyword matching degree is greater than or equal to a preset matching degree threshold, the corresponding subtext is determined to be a suspected risk subtext; or... When the matching degree of the negative keyword is less than the preset matching degree threshold, the corresponding subtext is determined to be a non-reviewed subtext; When the sentiment polarity result corresponding to the suspected risk subtext is negative, the suspected risk subtext is identified as a subtext to be reviewed; or, When the sentiment polarity result corresponding to the suspected risk subtext is non-negative, the corresponding subtext is determined to be a subtext not to be reviewed. The sub-text to be reviewed in the target text is marked to obtain the audio recognition result carrying the tag that needs to be framed for review.

4. The method according to claim 1, characterized in that, The step of performing frame extraction on the suspected video range set using a preset frame extraction strategy to obtain a set of keyframe images corresponding to the suspected video range includes: Obtain the video duration corresponding to the video to be detected; The number of segments to be reviewed is determined based on the video duration. The set of suspected video ranges was analyzed to obtain multiple sets of segment data to be reviewed. Based on preset selection rules, a target subset of segments to be reviewed, corresponding to the number of segments to be reviewed, is selected from the multiple sets of segment data to be reviewed. According to the preset frame extraction strategy, frame extraction is performed on each target segment to be reviewed in the target segment to be reviewed subset to obtain a key frame image subset corresponding to each target segment to be reviewed. The key frame image subset corresponding to each target segment to be reviewed contains at least two key frame images corresponding to each target segment to be reviewed. The keyframe image subsets corresponding to all target segments to be reviewed are aggregated to obtain the keyframe image set corresponding to the suspected video range.

5. The method according to claim 4, characterized in that, The step of performing frame extraction on each target segment to be reviewed in the target segment to be reviewed subset according to the preset frame extraction strategy, to obtain a keyframe image subset corresponding to each target segment to be reviewed, includes: Obtain the time range corresponding to each target segment to be reviewed in the subset of target segments to be reviewed; When the preset frame extraction strategy is the first frame extraction strategy, the start time point of the video to be detected and the end time point within the time range corresponding to each target segment to be reviewed are determined as the set of frame extraction time points corresponding to each target segment to be reviewed; or... When the preset frame extraction strategy is the second frame extraction strategy, the start time point, the middle time point and the end time point of the time range corresponding to each target segment to be reviewed are determined as the set of frame extraction time points corresponding to each target segment to be reviewed, and the middle time point is the midpoint between the start time point and the end time point. Based on the set of frame extraction time points corresponding to each target segment to be reviewed, the corresponding key image frames are extracted from the video to be detected, resulting in a key frame image subset corresponding to each target segment to be reviewed.

6. The method according to claim 1, characterized in that, The step of performing visual analysis on the keyframe image set, identifying negative content in the keyframe image set, and obtaining a target visual verification result corresponding to the keyframe image set includes: Obtain each keyframe image from the keyframe image set; Perform dual-path visual feature extraction on each keyframe image to obtain single-frame structured visual data corresponding to each keyframe image. The single-frame structured visual data includes single-frame visual semantic features and single-frame text sentiment determination results. The single-frame visual semantic features and single-frame text emotion judgment results corresponding to each keyframe image are fused to obtain the single-frame fused features corresponding to each keyframe image. Based on the temporal order of each frame in the keyframe image set, temporal features are extracted from the single-frame fusion features corresponding to each frame to obtain the global temporal features corresponding to the keyframe image set. The global temporal features are mapped and classified to obtain the visual verification result of the target.

7. The method according to claim 6, characterized in that, The step of performing dual-path visual feature extraction on each keyframe image to obtain single-frame structured visual data corresponding to each keyframe image includes: Based on the preset visual language model, visual semantic features are extracted from each keyframe image to obtain the visual semantic information corresponding to each keyframe image. Text extraction is performed on each keyframe image to obtain the extracted text corresponding to each keyframe image. Sentiment analysis and negative judgment are performed on the text extracted from the image corresponding to each keyframe image to obtain the sentiment judgment result of the text corresponding to each keyframe image. The visual semantic information corresponding to each keyframe image is structurally mapped to the text sentiment determination result corresponding to each keyframe image to obtain single-frame structured visual data corresponding to each keyframe image.

8. The method according to claim 1, characterized in that, The audio recognition result and the target visual verification result are fused and judged based on a preset fusion strategy to obtain a target judgment result for the video to be detected. The target judgment result is used to indicate whether the video to be detected contains negative information, including: Based on the target visual verification results, the negative confidence level of the visual modality is obtained; Based on the audio recognition results, the negative confidence score of the text modality is obtained. If the negative confidence score of the visual modality is greater than or equal to the first preset trigger threshold, or if the negative confidence score of the text modality is greater than or equal to the first preset trigger threshold, then the target judgment result is determined to be that the video to be detected is a video containing negative information; or... If the target negative confidence level is greater than or equal to the preset fusion judgment threshold, then the target judgment result is determined to confirm that the video to be detected is a video containing negative information; If the target negative confidence level is less than the preset fusion judgment threshold, then the target judgment result is determined to be that the video to be detected is a video that does not contain negative information.

9. A system for identifying negative information, characterized in that, The negative information identification system includes an electronic device, which includes a memory and a processor. The memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, it causes the negative information identification system to implement the negative information identification method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the negative information identification method as described in any one of claims 1-8.

Images