Media multi-mode content auditing method and system based on artificial intelligence technology

Through artificial intelligence technology combining object detection, face recognition and voice recognition, the problem of poor accuracy and inconsistent results of multimodal media content review is solved, efficient and accurate audit of multimodal content is achieved, and content security and quality.

CN119068399AInactive Publication Date: 2024-12-03GUANGDONG SOUTH SMART MEDIA TECH CO LTD

Patent Information

Application Number
CN202411561026.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-04
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When processing multimodal media content, the audit accuracy is poor and the results are inconsistent, so it is impossible to fully capture and analyze multiple information types in the video, resulting in reduced audit accuracy and inability to guarantee content security.

Method used

Using the multimodal media content review method based on artificial intelligence technology, we use the multimodal media content review method to receive content to be reviewed, judge the media type, separate audio and video, extract keyframes using gradient change detection, and combine target detection, face recognition and speech recognition models to analyze Violation information and output audit results through preset violation information database and natural language processing algorithm.

Benefits of technology

It realizes efficient and accurate audit of multimodal content, improves audit accuracy and consistency, and can comprehensively detect and identify illegal content, and is suitable for a variety of media content review scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068399A_ABST
    Figure CN119068399A_ABST
Patent Text Reader

Abstract

The invention relates to a media multi-mode content auditing method and system based on an artificial intelligence technology, and the method comprises the steps: receiving a media content to be audited, judging the media type, separating an audio from a video when the media content is judged to be the video, and extracting a key frame of the video through employing an algorithm based on gradient change detection; carrying out target detection on the key frame of the video by adopting a target detection model, carrying out face recognition on the key frame of the video by adopting a face recognition model, and carrying out text conversion processing on the audio by adopting a voice recognition model; and based on the target detection result, the face recognition result and the text conversion processing result, analyzing violation information through a preset violation information base and a natural language processing algorithm, and outputting an audit result. According to the method, comprehensive analysis of different media contents is realized by combining target detection, face recognition and voice recognition technologies, illegal contents can be efficiently and accurately detected and recognized, the auditing efficiency and accuracy are improved, and the method is suitable for various media content auditing scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent media, and in particular to a media multimodal content review method and system based on artificial intelligence technology. Background Art

[0002] With the development of technology, the forms of media content have become more diverse, including not only text, but also images, audio, video and other forms. These contents cover various fields from education and popular science to entertainment and leisure, showing the characteristics of diversity and massiveness. However, behind this rich and diverse media content, the review and management of media multimodal content faces huge challenges.

[0003] Most existing methods are aimed at auditing a single type of data, such as text content audit methods based on rule matching such as keyword filtering, which conducts matching checks on text by establishing a keyword library containing sensitive words and illegal words. Image content audit methods based on deep learning learn the characteristics of different types of images through a large amount of training data, so that they can be accurately identified. However, in the face of increasingly complex data forms, these methods are unable to fully capture and analyze the rich information of media multimodal content, limiting the depth and breadth of the audit.

[0004] Video is a multimodal content medium that integrates images, audio and text. It has the characteristics of intuitiveness and narrative, and has become one of the main forms of information dissemination. Traditional video content review methods mainly rely on manual proofreading. Manual review methods require reviewers to make judgments based on established standards, which are easily limited by subjective emotions and energy, and the review results are unstable, and consume a lot of manpower and time costs. This method seems to be unable to cope with the growing amount of data and multimodal content types. At present, although some video review models have achieved certain results in processing the review of a single data type, they still have significant limitations when facing multimodal information in videos. These models can often only process static images in videos, and cannot effectively identify illegal words in text, characters, and voice content in videos, or comprehensively analyze multiple types of information. This limitation reduces the accuracy of review and cannot fully guarantee the security and quality of content.

[0005] Therefore, there is an urgent need to build a comprehensive, efficient and intelligent media multimodal content review system to cope with the increasingly complex forms of media content, while ensuring the consistency and accuracy of the review and providing users with a safer and higher-quality content environment. Summary of the invention

[0006] The purpose of the present invention is to provide a media multimodal content review method and system based on artificial intelligence technology to solve the problems of poor review accuracy and inconsistent review results in the prior art.

[0007] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a media multimodal content review method based on artificial intelligence technology, the method comprising: Receive the media content to be reviewed and determine the media type. When the media content is determined to be a video, separate the audio from the video and extract the key frames of the video using an algorithm based on gradient change detection; The target detection model is used to detect targets in the key frames of the video, the face recognition model is used to recognize faces in the key frames of the video, and the speech recognition model is used to convert audio into text; Based on the target detection results, face recognition results and text conversion processing results, the preset violation information database and natural language processing algorithm are used to analyze the violation information and output the audit results.

[0008] As a further improvement of an embodiment of the present invention, the method further includes, when the media content is a picture, performing target detection and face recognition, and reviewing the target detection result and the face recognition result; When the media content is audio, the audio is converted into text and the converted text results are reviewed.

[0009] As a further improvement of an embodiment of the present invention, the method further includes: extracting the key frame of the video includes: Image processing technology is used to extract key frames from video content based on preset time intervals through inter-frame difference and gradient change detection algorithms.

[0010] As a further improvement of an embodiment of the present invention, the method further includes that the use of the target detection model to perform target detection on the key frames of the video includes: The key frame image is decomposed into multiple scale layers through a multi-scale feature pyramid to capture target information at different scales; Using a dynamic convolutional network to output the target information of different scales as a multi-scale feature map; Use temporal convolutional networks to extract temporal features from video frame sequences; Fusion of the multi-scale feature map with the time series feature to generate a comprehensive feature map containing time series information; Based on the comprehensive feature map, use the detection head to perform target detection and output the position, size, confidence and category label of each candidate box; Use the preset violation information library to output the violation information in the scenes and items in each candidate frame.

[0011] As a further improvement of an embodiment of the present invention, the method further includes that the face recognition of the key frames of the video using the face recognition model includes: The RetinaFace and FaceNet models are used to detect the face in the key frame. The specific steps include: Extract multi-scale features from key frame images based on the RetinaFace model to obtain multiple feature maps; Perform detection box regression on each position in each feature map to predict the face detection box; Use regression network to predict the key points of the face and determine the face area; Use the non-maximum suppression algorithm to remove duplicate detection frames, retain the detection frame with the highest confidence, and send the face area image to the FaceNet model; Extract high-dimensional features of face area images based on the FaceNet model and generate feature vectors; By comparing the feature vector with the pre-stored feature vector, the identity of the face is determined and the identity information of the face is output.

[0012] As a further improvement of an embodiment of the present invention, the method further includes: the step of using a face recognition model to perform face recognition on key frames of a video further includes: For multiple face images of the same person, the scenes of the face images are classified, and the feature vectors of the face images in different scenes are weighted averagely to generate the final feature vector of the face.

[0013] As a further improvement of an embodiment of the present invention, the method further includes that the audio-to-text conversion process using a speech recognition model includes: The Whisper model is used to convert audio into text. The specific steps include: Filter the audio through a filter to generate a spectrogram, and extract features from the spectrogram; Extract high-level audio features through the encoder; The decoder converts high-level audio features into text and performs text error correction to optimize the audio-to-text conversion result. The steps to identify the converted text violations include: Use the LAC tool to perform word segmentation on the converted text, splitting the text into vocabulary and part-of-speech tags; Use the preset violation information database and natural language processing algorithm to traverse the word segmentation results and output the frequency of occurrence of the violation words and the location information of the text.

[0014] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a media multimodal content review system based on artificial intelligence technology, the system comprising a judgment unit, a detection module and a review module; The determination module is used to receive the media content to be reviewed and determine the media type. When the media content is determined to be a video, the audio is separated from the video, and a key frame of the video is extracted using an algorithm based on gradient change detection. The detection module is used to perform target detection on the key frames of the video using a target detection model, perform face recognition on the key frames of the video using a face recognition model, and perform text conversion processing on the audio using a speech recognition model; The audit module is used to analyze violation information and output audit results based on target detection results, face recognition results and text conversion processing results through a preset violation information database and natural language processing algorithm.

[0015] In order to achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor, characterized in that the memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the media multimodal content review method based on artificial intelligence technology as described above are implemented.

[0016] In order to achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, the steps in the media multimodal content review method based on artificial intelligence technology as described above are implemented.

[0017] Compared with the prior art, the present invention provides a media multimodal content review method and system based on artificial intelligence technology, which realizes comprehensive analysis of different media content by combining target detection, face recognition and voice recognition technology, can efficiently and accurately detect and identify illegal content, improve review efficiency and accuracy, and is suitable for a variety of media content review scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an overall flow chart of the media multimodal content review method based on artificial intelligence technology described in the present invention.

[0019] Figure 2 It is a schematic diagram of the architecture of the media multimodal content review system based on artificial intelligence technology described in the present invention. DETAILED DESCRIPTION

[0020] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0021] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0022] In the first embodiment of the present invention, the present invention provides a media multimodal content review method based on artificial intelligence technology, such as Figure 1 As shown, the method includes, S1: receiving the media content to be reviewed and determining the media type. When the media content is determined to be a video, the audio and video are separated, and a key frame of the video is extracted using an algorithm based on gradient change detection; S2: Use the target detection model to detect targets on the key frames of the video, use the face recognition model to recognize faces on the key frames of the video, and use the speech recognition model to convert audio into text; S3: Based on the target detection results, face recognition results and text conversion processing results, the preset violation information database and natural language processing algorithm are used to analyze the violation information and output the audit results.

[0023] Furthermore, when the media content is determined to be a picture, target detection and face recognition are performed, and the target detection results and face recognition results are reviewed; when the media content is determined to be audio, audio-to-text processing is performed, and the converted text results are reviewed.

[0024] In a specific embodiment of the present invention, the key frames of the video are extracted by using image processing technology, through frame difference and gradient change detection algorithm, to extract the key frames from the video content according to a preset time interval.

[0025] Furthermore, the OpenCV library is used for image processing to identify key frames by calculating the difference between consecutive frames and the image gradient changes. First, the video stream is read and segmented into a single-frame image sequence; then, the inter-frame difference technology is used to detect the motion area and the significantly changed part by comparing the change in the pixel values ​​of adjacent frames; at the same time, the gradient change detection algorithm is applied to analyze the edge information of each frame image to identify the area with significant changes. Finally, based on the preset time interval and the degree of change detected, the representative key frame image is extracted. In this process, OpenCV provides efficient image processing functions and tools to support frame reading, difference calculation, gradient calculation, and key frame extraction, making the whole process efficient and accurate.

[0026] In a specific embodiment of the present invention, the object detection model is used to perform object detection on the key frames of the video, specifically, The key frame image is decomposed into multiple scale layers through a multi-scale feature pyramid to capture target information at different scales; Using a dynamic convolutional network to output the target information of different scales as a multi-scale feature map; Use temporal convolutional networks to extract temporal features from video frame sequences; Fusion of the multi-scale feature map with the time series feature to generate a comprehensive feature map containing time series information; Based on the comprehensive feature map, use the detection head to perform target detection and output the position, size, confidence and category label of each candidate box; Use the preset violation information library to output the violation information in the scenes and items in each candidate frame.

[0027] Furthermore, in a specific embodiment of the present invention, a multi-scale dynamic detection model is used to detect objects in key frames of a video. The model combines a multi-scale feature pyramid and a dynamic convolutional network, and can adaptively adjust the parameters of the convolution kernel to cope with object detection of different scales and complexities.

[0028] The specific steps include: Multi-scale feature pyramid: By constructing a multi-scale feature pyramid, the input key frame image is decomposed into multiple scale layers, each of which contains feature maps of different resolutions. These feature maps are used to capture target information of different scales.

[0029] Dynamic convolutional network: In the feature extraction stage, a dynamic convolutional network is used. This network introduces adaptive convolution kernel parameters and dynamically adjusts the convolution kernel according to the content of the input feature map to improve the detection ability of complex backgrounds and targets.

[0030] Feature fusion: The feature maps of different scale layers are fused to generate a multi-scale comprehensive feature map. This feature map can capture the information of large and small targets at the same time, improving the robustness of detection.

[0031] Object detection: Based on the comprehensive feature map, the detection head is used for object detection. The detection head consists of two parts: position regression and category classification, which can output the position, size, confidence and category label of each candidate box.

[0032] Adaptive threshold adjustment: Introduce an adaptive threshold adjustment mechanism to dynamically adjust the detection threshold according to the detection confidence of the target to reduce false detections and missed detections.

[0033] Furthermore, in the target detection process, in addition to the static features of a single frame, a temporal information fusion mechanism is introduced to enhance the target detection accuracy of key frames by combining the information of previous and next frames.

[0034] The specific steps include: Temporal feature extraction: Temporal features are extracted from video frame sequences through long short-term memory networks (LSTM) or temporal convolutional networks (TCN). These features are used to describe the temporal change trend of the target.

[0035] Temporal feature fusion: The extracted temporal features are fused with the multi-scale feature map of the current key frame to generate a comprehensive feature map containing temporal information.

[0036] Temporal consistency detection: Based on the comprehensive feature map, the detection head is used to detect the target, and the temporal consistency of the target is checked through the temporal consistency detection module, and the detection results that do not conform to the temporal changes are filtered out.

[0037] Furthermore, a self-supervised learning mechanism is introduced to optimize the target detection model and improve the model's adaptability and generalization performance.

[0038] The specific steps include: Self-supervised feature learning: During the model training phase, the learning ability of the feature extraction module is enhanced by designing self-supervised tasks (such as predicting the content of occluded areas or inter-frame changes), so that it can better understand image content and temporal changes.

[0039] Self-supervised optimization: During the target detection process, the model parameters are dynamically adjusted through the online learning mechanism according to the detection results and the feedback of the self-supervised tasks to continuously optimize the target detection performance.

[0040] In a specific embodiment of the present invention, a face recognition model is used to perform face recognition on key frames of a video, specifically, The RetinaFace and FaceNet models are used to detect the face in the key frame. The specific steps include: Extract multi-scale features from key frame images based on the RetinaFace model to obtain multiple feature maps; Perform detection box regression on each position in each feature map to predict the face detection box; Use regression network to predict the key points of the face and determine the face area; Use the non-maximum suppression algorithm to remove duplicate detection frames, retain the detection frame with the highest confidence, and send the face area image to the FaceNet model; Extract high-dimensional features of face area images based on the FaceNet model and generate feature vectors; By comparing the feature vector with the pre-stored feature vector, the identity of the face is determined and the identity information of the face is output.

[0041] Furthermore, the RetinaFace model includes: Image input: Receive the original image.

[0042] Feature extraction: Using MobileNet with deep separable convolution as the backbone network, the multi-scale features of the image are extracted through efficient convolution operations, thereby optimizing computing performance. Combined with FPN, it can process faces of different sizes and improve detection accuracy.

[0043] Bounding box prediction: Perform bounding box regression for each position in each feature map to predict the bounding box of the face, including position, size, and pose information, to accurately locate the face area.

[0044] Key point positioning: The key points of the face (such as eyes, nose, and mouth) are predicted through a regression network to further accurately locate and adjust the face area.

[0045] Result optimization: Apply the NMS algorithm to remove duplicate detection frames and retain the detection frames with the highest confidence to ensure the accuracy and clarity of the detection results.

[0046] Furthermore, the FaceNet model includes: Image input: Receives an accurate face region image extracted from the RetinaFace model.

[0047] Feature extraction: Use MobileNet with depthwise separable convolution to extract features from the input face image. Through multiple convolutional and pooling layers, MobileNet maps the image to a high-dimensional feature space to extract key features.

[0048] Feature embedding: The output of the convolutional network is converted into a 128-dimensional embedding vector through a fully connected layer, and the features of each face are mapped into a 128-dimensional feature space to generate a feature vector representation.

[0049] Similarity evaluation: Calculate the Euclidean distance between two 128-dimensional feature vectors to evaluate the similarity between faces. The smaller the distance, the higher the similarity between the two faces.

[0050] Identification and verification: The identity of the face is determined by comparing the feature vector with the known feature vectors in the database. The identification and verification process is completed by judging whether they belong to the same person based on the set distance threshold.

[0051] In a specific embodiment of the present invention, using a face recognition model to perform face recognition on key frames of a video also includes: For multiple face images of the same person, the scenes of the face images are classified, and the feature vectors of the face images in different scenes are weighted averagely to generate the final feature vector of the face.

[0052] It should be noted that when there are multiple photos of the person, the photos can be first classified into scenes, such as: distant view, close-up, multiple people, single person, front face, side face, etc., and then the person vectors in each different scene are weighted averaged separately. The person can finally obtain face vector data in different scenes, which is conducive to improving the accuracy in actual detection and the robustness of the model.

[0053] In a specific embodiment of the present invention, a speech recognition model is used to convert audio into text, specifically, The Whisper model is used to convert audio into text. The specific steps include: Filter the audio through a filter to generate a spectrogram, and extract features from the spectrogram; Extract high-level audio features through the encoder; The decoder converts high-level audio features into text and performs text error correction to optimize the audio-to-text conversion results.

[0054] Furthermore, the separated raw audio data is received, which can support mono or stereo, and is usually processed at a sampling rate of 16kHz or higher to ensure sound quality. The raw audio signal is converted into Mel Frequency Cepstral Coefficient (MFCC) features to capture the spectral characteristics in the audio. The audio signal is filtered using a Mel filter to generate a Mel spectrogram, from which the MFCC features are extracted.

[0055] Furthermore, by using CNN and long short-term memory network in the encoder to extract high-level representations from audio features, the audio features are converted into deep feature representations that the model can handle. In addition, the Transformer structure is used in the decoder to convert the high-level representation generated by the encoder into text, and the self-attention mechanism is used to capture the long-term dependencies in the audio sequence to improve the context understanding ability.

[0056] Furthermore, in the text output stage, the Whisper model combines contextual information to perform text error correction, optimize the accuracy of audio-to-text conversion results, correct possible recognition errors, and provide more accurate text data.

[0057] In a specific embodiment of the present invention, the step of identifying the converted text violation information is specifically, Use the LAC tool to perform word segmentation on the converted text, splitting the text into vocabulary and part-of-speech tags; Use the preset violation information database and natural language processing algorithm to traverse the word segmentation results and output the frequency of occurrence of the violation words and the location information of the text.

[0058] Furthermore, OpenCC is used to convert Traditional Chinese into Simplified Chinese to ensure the consistency of the text and the accuracy of the processing. The converted text is segmented by the LAC (Lexical Analysis of Chinese) Chinese word segmentation tool to split the text into more fine-grained words. By loading a custom word library, LAC can improve the sensitivity and accurate recognition of domain-specific terms. The word segmentation results include each word and its part-of-speech tag, and are stored in a list.

[0059] Furthermore, the system traverses the word segmentation results and checks whether each word exists in the custom illegal word library. If it exists, the number of occurrences of the word and the text paragraph in which it is located are recorded. The statistical results of sensitive word matching include the vocabulary of the sensitive word, the number of occurrences and the specific location in the text.

[0060] Furthermore, the system summarizes all the identified violating words and their occurrence times into a final report. The report lists in detail the frequency of each sensitive word and the text paragraph in which it appears. The statistical results provide a basis for subsequent content review and analysis to ensure that the media content meets the preset security and compliance standards.

[0061] In the second embodiment of the present invention, the present invention provides a media multimodal content review system based on artificial intelligence technology, such as Figure 2 As shown, the system includes a judgment module 100, a detection module 200 and an audit module 300; The determination module 100 is used to receive the media content to be reviewed and determine the media type. When the media content is determined to be a video, the audio is separated from the video and a key frame of the video is extracted using an algorithm based on gradient change detection. The detection module 200 is used to perform target detection on key frames of the video using a target detection model, perform face recognition on key frames of the video using a face recognition model, and perform text conversion processing on audio using a speech recognition model; The audit module 300 is used to analyze violation information and output audit results based on target detection results, face recognition results and text conversion processing results through a preset violation information database and natural language processing algorithm.

[0062] Furthermore, the detection module 200 includes an image detection module 201 , a face detection module 202 and a text detection module 203 .

[0063] Furthermore, the image detection module 201 is used to conduct a comprehensive review of the received image data or the key frame images obtained by frame extraction. According to the preset violation categories, the module performs target detection on the scenes and objects in the image to accurately identify various types of content in the image (such as QR codes, LOGOs, etc.), and determine whether there is any violation information (such as sensitive content, WeChat business advertisements, bad information, etc.). In addition, the module also performs quality inspection on the aesthetics and clarity of the image, and provides detailed review results to ensure that the image content complies with relevant regulations and standards.

[0064] Furthermore, the face detection module 202 is used to detect and identify faces in the received images or key frame images obtained by frame extraction, lock the specific face area and match its corresponding name. In particular, for specific people, the system uses customized data sets for special training to achieve high-accuracy facial recognition and ensure the rigor, accuracy and authority of media content.

[0065] Furthermore, the text detection module 203 uses optical character recognition (OCR) technology to extract text information from pictures and video frames, identifies all text by scanning each area, including titles, descriptions, advertisements and logos, and converts them into editable text data. The module applies classification rules to detect whether it contains sensitive, advertising promotion, vulgar insults or low-quality spam and other illegal content, and combines NLP natural language processing technology to extract keywords and generate a detailed audit report.

[0066] In a third embodiment of the present invention, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the media multimodal content review method based on artificial intelligence technology as described above are implemented.

[0067] In a fourth embodiment of the present invention, the present invention provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps in the media multimodal content review method based on artificial intelligence technology as described above.

[0068] To sum up, the present invention provides a media multimodal content review method and system based on artificial intelligence technology, which realizes comprehensive analysis of different media content by combining target detection, face recognition and voice recognition technology, can efficiently and accurately detect and identify illegal content, improve review efficiency and accuracy, and is suitable for a variety of media content review scenarios.

[0069] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the modules described above can refer to the corresponding process in the aforementioned method implementation, and will not be repeated here.

[0071] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0072] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0073] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for a computer system (which can be a personal computer, a server, or a network system, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0074] Finally, it should be noted that the above implementation modes are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned implementation modes, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned implementation modes, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various implementation modes of the present application.

Claims

1. A media multimodal content review method based on artificial intelligence technology, characterized by: include, Receive the media content to be reviewed and determine the media type. When the media content is determined to be a video, separate the audio from the video and extract the key frames of the video using an algorithm based on gradient change detection; The target detection model is used to detect targets in the key frames of the video, the face recognition model is used to recognize faces in the key frames of the video, and the speech recognition model is used to convert audio into text; Based on the target detection results, face recognition results and text conversion processing results, the preset violation information database and natural language processing algorithm are used to analyze the violation information and output the audit results.

2. According to claim 1, the method for media multimodal content review based on artificial intelligence technology is characterized by: Also includes, When the media content is determined to be a picture, target detection and face recognition are performed, and the target detection results and face recognition results are reviewed; When the media content is determined to be audio, the audio is converted into text and the converted text result is reviewed.

3. The method for media multimodal content review based on artificial intelligence technology according to claim 1 is characterized in that: The key frames of the extracted video include: Image processing technology is used to extract key frames from video content at preset time intervals through frame difference and gradient change detection algorithms.

4. The method for reviewing media multimodal content based on artificial intelligence technology according to claim 3 is characterized by: The target detection model is used to detect the target of the key frame of the video, including: The key frame image is decomposed into multiple scale layers through a multi-scale feature pyramid to capture target information at different scales; Using a dynamic convolutional network to output the target information of different scales as a multi-scale feature map; Use temporal convolutional networks to extract temporal features from video frame sequences; Fusion of the multi-scale feature map with the time series feature to generate a comprehensive feature map containing time series information; Based on the comprehensive feature map, use the detection head to perform target detection and output the position, size, confidence and category label of each candidate box; Use the preset violation information library to output the violation information in the scenes and items in each candidate frame.

5. The method for media multimodal content review based on artificial intelligence technology according to claim 3 is characterized by: The face recognition model is used to perform face recognition on the key frames of the video, including: The RetinaFace and FaceNet models are used to detect the face in the key frame. The specific steps include: Extract multi-scale features from key frame images based on the RetinaFace model to obtain multiple feature maps; Perform detection box regression on each position in each feature map to predict the face detection box; Use regression network to predict the key points of the face and determine the face area; Use the non-maximum suppression algorithm to remove duplicate detection frames, retain the detection frame with the highest confidence, and send the face area image to the FaceNet model; Extract high-dimensional features of face area images based on the FaceNet model and generate feature vectors; By comparing the feature vector with the pre-stored feature vector, the identity of the face is determined and the identity information of the face is output.

6. The method for reviewing multimodal content of media based on artificial intelligence technology according to claim 5 is characterized by: The face recognition model is used to perform face recognition on the key frames of the video, and further includes: For multiple face images of the same person, the scenes of the face images are classified, and the feature vectors of the face images in different scenes are weighted averagely to generate the final feature vector of the face.

7. The method for reviewing media multimodal content based on artificial intelligence technology according to claim 1 is characterized in that: The audio-to-text conversion process using a speech recognition model includes: The Whisper model is used to convert audio into text. The specific steps include: Filter the audio through a filter to generate a spectrogram, and extract features from the spectrogram; Extract high-level audio features through the encoder; The decoder converts high-level audio features into text and performs text error correction to optimize the audio-to-text conversion result. The steps to identify the converted text violations include: Use the LAC tool to perform word segmentation on the converted text, splitting the text into vocabulary and part-of-speech tags; Use the preset violation information database and natural language processing algorithm to traverse the word segmentation results and output the frequency of occurrence of the violation words and the location information of the text.

8. A media multimodal content review system based on artificial intelligence technology, characterized by: It includes judgment module, detection module and audit module; The determination module is used to receive the media content to be reviewed and determine the media type. When the media content is determined to be a video, the audio is separated from the video, and a key frame of the video is extracted using an algorithm based on gradient change detection. The detection module is used to perform target detection on the key frames of the video using a target detection model, perform face recognition on the key frames of the video using a face recognition model, and perform text conversion processing on the audio using a speech recognition model; The audit module is used to analyze violation information and output audit results based on target detection results, face recognition results and text conversion processing results through a preset violation information database and natural language processing algorithm.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the media multimodal content review method based on artificial intelligence technology as described in any one of claims 1-7 are implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps in the media multimodal content review method based on artificial intelligence technology as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • An automatic driving visual perception optimization method based on characteristic time sequence correlation

    CN109934096A

  • Video anomaly detection method and system based on multiple branches

    CN113033424A

  • Video target detection system and method based on multi-level spatial-temporal feature fusion

    CN113642498A

  • Product display video automatic auditing method and system

    CN113920085A

  • Video processing method, device and equipment and computer storage medium

    CN114972087A

Cited By

  • Digital media content processing method, device and system

    CN119420970A

  • Short video element analysis method and system

    CN120106081A

  • Method and system for detecting materials before casting

    CN120302085A

  • Multi-modal content review method and device, medium and computer program

    CN120493147A

  • Intelligent auditing method and device based on target detection and multi-modal large model

    CN120561323A