Live content intelligent monitoring management system and method based on deep learning

Abstract

The invention provides a live broadcast content intelligent monitoring management system and method based on deep learning, and relates to the technical field of video live broadcast monitoring, and the method comprises the steps: collecting live broadcast content information in real time; performing extraction processing to obtain a motion capture image set and a detail recognition image set; constructing an intelligent monitoring network; respectively inputting the motion capture image set and the detail identification image set into an intelligent monitoring network to obtain a video analysis result; respectively inputting the audio information set and the text information set into an intelligent monitoring network to obtain an audio analysis result and a text analysis result; and violation judgment and alarm processing are carried out. Through the method and the device, the technical problem of low auditing efficiency caused by the fact that the live broadcast content relates to various forms, the content quantity is large and the real-time performance is high in the prior art can be solved, and the technical effect of improving the content auditing efficiency and accuracy is achieved by constructing a multi-channel parallel monitoring auditing mechanism to realize omnibearing and real-time monitoring of the live broadcast content.

Description

Technical Field

[0001] The present application relates to the field of live video monitoring technology, and in particular to a live content intelligent monitoring management system and method based on deep learning. Background Art

[0002] With the development of internet technology and the prevalence of smart devices, live streaming has become an important form of entertainment and information dissemination in people's daily lives. However, the diversity and real-time nature of live streaming content pose significant challenges to content regulation. Traditional manual review methods are no longer able to meet these demands, primarily due to low review efficiency and subjective judgments that can easily lead to misjudgments. To address these issues, some technologies have attempted to introduce automated content review mechanisms, such as rule-based filtering systems or simple image recognition technologies. However, these methods are often only able to handle a single type of content, such as video or text only, and are ineffective for complex scenarios and dynamically changing live content. Furthermore, the lack of deep learning and artificial intelligence technologies means that review efficiency and accuracy remain to be improved.

[0003] In summary, the existing technology has a technical problem of low review efficiency due to the fact that live broadcast content involves various forms, large content volume and strong real-time nature. Summary of the Invention

[0004] The purpose of this application is to provide a live broadcast content intelligent monitoring and management system and method based on deep learning, in order to solve the technical problem in the prior art that the live broadcast content involves multiple forms, large content volume and strong real-time nature, resulting in low review efficiency.

[0005] In view of the above problems, this application provides a live content intelligent monitoring and management system and method based on deep learning.

[0006] In the first aspect, the present application provides a live content intelligent monitoring and management system based on deep learning, wherein the live content intelligent monitoring and management system based on deep learning includes: a data acquisition module, the data acquisition module is used to collect live content information in real time, and obtain a content information collection, wherein the content information collection includes a video information collection, an audio information collection and a text information collection; a video processing module, the video processing module is used to extract and process the video information collection to obtain a motion capture image collection and a detail recognition image collection; a monitoring network construction module, the monitoring network construction module is used to construct an intelligent monitoring network, and the intelligent monitoring network includes a video monitoring channel, an audio monitoring channel and a text monitoring channel. Channel, wherein the video monitoring channel includes a motion capture sub-channel and a detail recognition sub-channel; a video analysis module, the video analysis module is used to input the motion capture image collection and the detail recognition image collection into the motion capture sub-channel and the detail recognition sub-channel of the intelligent monitoring network respectively, to obtain video analysis results; a text analysis module, the text analysis module is used to input the audio information collection and the text information collection into the audio monitoring channel and the text monitoring channel of the intelligent monitoring network respectively, to obtain audio analysis results and text analysis results; a violation determination module, the violation determination module is used to perform violation determination and alarm processing based on the video analysis results, audio analysis results and text analysis results.

[0007] In the second aspect, the present application also provides an intelligent monitoring and management method for live broadcast content based on deep learning, wherein the intelligent monitoring and management method for live broadcast content based on deep learning includes: real-time collection of live broadcast content information to obtain a content information collection, wherein the content information collection includes a video information collection, an audio information collection and a text information collection; extracting and processing the video information collection to obtain a motion capture image collection and a detail recognition image collection; constructing an intelligent monitoring network, wherein the intelligent monitoring network includes a video monitoring channel, an audio monitoring channel and a text monitoring channel, wherein the video monitoring channel includes a motion capture sub-channel and a detail recognition sub-channel; inputting the motion capture image collection and the detail recognition image collection into the motion capture sub-channel and the detail recognition sub-channel of the intelligent monitoring network respectively to obtain video analysis results; inputting the audio information collection and the text information collection into the audio monitoring channel and the text monitoring channel of the intelligent monitoring network respectively to obtain audio analysis results and text analysis results; based on the video analysis results, audio analysis results and text analysis results, respectively performing violation judgment and alarm processing.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] The data acquisition module is used to collect live content information in real time to obtain a content information collection, wherein the content information collection includes a video information collection, an audio information collection, and a text information collection; the video processing module is used to extract and process the video information collection to obtain a motion capture image collection and a detail recognition image collection; the monitoring network construction module is used to build an intelligent monitoring network, wherein the intelligent monitoring network includes a video monitoring channel, an audio monitoring channel, and a text monitoring channel, wherein the video monitoring channel includes a motion capture sub-channel and a detail recognition sub-channel; the video analysis module is used to input the motion capture image collection and the detail recognition image collection into the motion capture sub-channel and the detail recognition sub-channel of the intelligent monitoring network respectively to obtain video analysis results; the text analysis module is used to input the audio information collection and the text information collection into the audio monitoring channel and the text monitoring channel of the intelligent monitoring network respectively to obtain audio analysis results and text analysis results; the violation determination module is used to perform violation determination and alarm processing based on the video analysis results, audio analysis results, and text analysis results. In other words, by building a multi-channel parallel monitoring and review mechanism, all-round and real-time monitoring of live broadcast content is achieved, achieving the technical effect of improving the efficiency and accuracy of content review.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the description, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically listed below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in this application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and a person of ordinary skill in the art can obtain other drawings based on the provided drawings without creative work.

[0012] Figure 1 This is a structural diagram of the live content intelligent monitoring and management system based on deep learning in this application;

[0013] Figure 2 This is a flowchart of the deep learning-based intelligent monitoring and management method for live content in this application.

[0014] Explanation of the accompanying symbols: data acquisition module 11, video processing module 12, monitoring network construction module 13, video analysis module 14, text analysis module 15, violation determination module 16. DETAILED DESCRIPTION

[0015] This application provides a deep learning-based intelligent monitoring and management system for live content, addressing the existing technical issues of low review efficiency due to the diverse nature of live content, its large volume, and its strong real-time nature. By establishing a multi-channel parallel monitoring and review mechanism, it achieves comprehensive, real-time monitoring of live content, achieving the technical effect of improving content review efficiency and accuracy.

[0016] Below, the technical solutions in this application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, rather than all the embodiments of this application. It should be understood that this application is not limited to the example embodiments described herein. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. It should also be noted that, for the convenience of description, only the parts related to this application, rather than all of them, are shown in the accompanying drawings.

[0017] For example, see the attached Figure 1 The present application provides a live content intelligent monitoring and management system based on deep learning, wherein the live content intelligent monitoring and management system based on deep learning is used to implement the steps of the live content intelligent monitoring and management method based on deep learning, and the live content intelligent monitoring and management system based on deep learning includes:

[0018] The data collection module 11 is used to collect live content information in real time and obtain a content information collection, wherein the content information collection includes a video information collection, an audio information collection and a text information collection.

[0019] Specifically, use a video capture device to obtain the live video stream in real time, use a microphone or other audio capture device to obtain the sound in the live broadcast in real time, and capture text information from the chat room, barrage system or other interactive interface of the live broadcast platform through API calls or web crawler technology. Video information includes picture content, character movements, etc.; audio information involves voice, background music, etc.; text information includes barrage, chat content, etc. The collected video stream is packaged at a certain time interval (such as every second or every few frames) to form a video information collection. The collected audio stream is segmented to form an audio information collection. The text information captured in real time is summarized in chronological order to form a text information collection. By collecting this information in real time, a variety of live broadcast information can be comprehensively obtained.

[0020] The video processing module 12 is used to extract and process the video information collection to obtain a motion capture image collection and a detail recognition image collection.

[0021] Specifically, video images are extracted frame by frame from the video information collection, the extracted frames are resized to a uniform size, and the video images are subjected to denoising, contrast enhancement, and other steps to improve image quality. Everyday images are collected to train the convolutional neural network model, resulting in a semantic segmentation model. The video images from the video information collection are input into the semantic segmentation model for semantic segmentation, resulting in a set of segmented images. The image data in the segmented image set is downsampled to extract the dynamic portion of the image, and the key frames of the image data are filtered to extract the static portion of the image, resulting in a collection of motion capture images and a collection of detail recognition images. By extracting key motion capture images and detail recognition images, the actions and static details in the video can be more accurately identified and captured, improving the ability to recognize and understand actions and details in the video content.

[0022] The monitoring network construction module 13 is used to construct an intelligent monitoring network, which includes a video monitoring channel, an audio monitoring channel and a text monitoring channel, wherein the video monitoring channel includes a motion capture sub-channel and a detail recognition sub-channel.

[0023] Specifically, video data, audio data, and text data containing both normal and illegal content are collected and annotated to distinguish between normal and illegal content. The video data, audio data, and text data are integrated into a video surveillance training dataset, an audio surveillance training dataset, and a text surveillance training dataset. The video surveillance training dataset includes a motion capture image training set and a detail recognition image training set. Supervised training is performed on each of the three datasets to obtain a video surveillance channel, an audio surveillance channel, and a text surveillance channel. The motion capture image training set and the detail recognition image training set are used to train a convolutional neural network (CNN) or other deep learning model to obtain a motion capture sub-channel and a detail recognition sub-channel; the audio surveillance training dataset is used to train an audio surveillance channel; and the text surveillance training dataset is used for a natural language processing (NLP) model to obtain a text surveillance channel. The three channels are connected in parallel to form an intelligent surveillance network. By connecting different channels in parallel, video, audio, and text content can be processed and analyzed simultaneously, thereby improving the accuracy and efficiency of surveillance.

[0024] The video analysis module 14 is used to input the motion capture image collection and the detail recognition image collection into the motion capture sub-channel and the detail recognition sub-channel of the intelligent monitoring network respectively to obtain video analysis results.

[0025] Specifically, the motion capture image collection contains images of changes in human motion, and the detail recognition image collection contains images of scene details. The motion capture image collection and detail recognition image collection are respectively input into the motion capture sub-channel and detail recognition sub-channel of the intelligent monitoring network for feature extraction, recognition and classification of actions in the image, including running, jumping, falling, and other actions; and recognition of details in the image, including faces, specific objects, etc. The analysis results of the motion capture sub-channel provide detailed information about the actions in the video, and the analysis results of the detail recognition sub-channel provide detailed information about the details in the video. By inputting and analyzing the motion capture image collection and detail recognition image collection, a deep understanding of the video content, including its structure and composition, is achieved. Through processing by the intelligent monitoring network, the video content is monitored, including the recognition and analysis of actions and details in the video.

[0026] The text analysis module 15 is used to input the audio information collection and the text information collection into the audio monitoring channel and the text monitoring channel of the intelligent monitoring network respectively to obtain audio analysis results and text analysis results.

[0027] Specifically, the audio and text information collections are input into the audio and text monitoring channels of the intelligent monitoring network, respectively. The audio content, including violent language and sensitive vocabulary, is identified to generate audio analysis results; the text content, including inappropriate speech and rumors, is identified to generate audio analysis results. The analysis results of the audio monitoring channel provide detailed information about the audio content, while the analysis results of the text monitoring channel provide detailed information about the text content. By inputting and analyzing the audio and text information collections, a deep understanding of the audio and text content, including their structure and composition, is achieved, enabling accurate and efficient monitoring of audio and text content, including the identification and analysis of sounds and intonations in the audio and illegal information in the text.

[0028] The violation determination module 16 is used to perform violation determination and alarm processing based on the video analysis results, audio analysis results and text analysis results.

[0029] Specifically, historical violation records are collected and clustered to obtain a violation level comparison table. The video analysis results, audio analysis results, and text analysis results are matched with the violation level comparison table, and the corresponding violation level is matched to each analysis result. Based on the matched violation level, it is determined whether it is a violation. Once it is determined that the content violates specific standards, alarm processing is triggered. Alarm processing includes recording the violation incident, notifying relevant personnel for review, automatically taking measures (such as banning, blocking accounts, etc.), and reporting to regulatory agencies. Alarm processing ensures timely response and processing of illegal content, improving the efficiency and security of content monitoring.

[0030] Furthermore, the video processing module 12 in the live content intelligent monitoring and management system based on deep learning is also used to:

[0031] Constructing a semantic segmentation model; inputting the video information collection into the semantic segmentation model for semantic segmentation to obtain a segmented image collection; and processing based on the segmented image collection to obtain a motion capture image collection and a detail recognition image collection.

[0032] Specifically, everyday images are collected from multiple sources and segmented to distinguish and segment the human body from the background. The acquired and segmented images are divided into training, validation, and test sets, which are then used to train, validate, and test a convolutional neural network model. Ultimately, a semantic segmentation model is constructed for image segmentation. Individual video frames are extracted from the resulting video information collection and preprocessed, such as resizing and normalizing, to meet model input requirements. The preprocessed video frames are input into the trained semantic segmentation model, which outputs a classification label for each pixel, forming a collection of segmented images. Segmented images are obtained for each frame in the video, meaning that each pixel is labeled as belonging to a specific category. The image data in the segmented image collection is downsampled by reducing the image resolution or the number of frames to extract the dynamic portions of the image, resulting in a collection of motion capture images, including human motion. Key frames in the segmented image collection are filtered to extract static regions, generating a collection of detail recognition images, including static scenes or object details. The acquisition of motion capture image collections and detail recognition image collections enables more accurate recognition and capture of dynamic changes, static scenes or object details in videos.

[0033] Furthermore, the video processing module 12 in the live content intelligent monitoring and management system based on deep learning is also used to:

[0034] Based on big data, multiple daily life images are collected to obtain multiple sample image information sets; based on the differences between human body features and background features, the image information in the multiple sample image information sets is segmented to obtain multiple sample image segmentation result sets; the multiple sample image information sets and the multiple sample image segmentation result sets are divided to obtain training data sets and verification data sets; based on the training data sets and verification data sets, a convolutional neural network model is supervised trained, verified and tested to obtain the semantic segmentation model.

[0035] Specifically, use big data technology to collect a large number of daily life images from multiple sources. Include various scenes, such as home environments, streets, public places, etc., to reflect the diversity of daily life. Ensure that the sample images cover different environments, lighting conditions, human postures and background types to enhance the generalization ability of the model. Analyze the human features (such as skin color, shape, posture, etc.) and background features (such as color, texture, scene layout, etc.) in the image. Segment the image information based on human features and background features. Image segmentation is a technology that divides an image into multiple parts or regions, each region representing a different object or background. Apply image processing techniques such as threshold segmentation, edge detection, region growing, etc., or deep learning algorithms such as Mask R-CNN, U-Net, etc., to distinguish and segment the human body and background in the image.

[0036] The collected and processed sample image information and sample image segmentation results are randomly divided into two parts: a training dataset and a validation dataset. The training dataset is used to train the machine learning model, while the validation dataset is used to evaluate the model's performance and generalization ability. A common split is 70% training data and 30% validation data, but the specific ratio can be adjusted based on the actual situation. Select a suitable convolutional neural network architecture, such as FCN (fully convolutional network), U-Net, SegNet, or PSPNet, four of the most widely used deep learning models in image processing, particularly image segmentation. The convolutional neural network model is trained using the training dataset for supervised training, using labeled input images and corresponding outputs (e.g., image segmentation masks). During training, the model continuously adjusts its parameters to minimize the discrepancy between predicted and actual outputs. After training, the validation dataset is used to evaluate the model's performance and make any necessary adjustments. A portion of the data not used in training and validation is typically retained as a test set for final evaluation of the model to objectively assess its generalization ability. The result is a model that can classify each pixel in the image, known as a semantic segmentation model, which is commonly used for image segmentation tasks. Improve the efficiency of model training by reasonably dividing the data set and using efficient computing resources.

[0037] Furthermore, the video processing module 12 in the live content intelligent monitoring and management system based on deep learning is also used to:

[0038] The image data in the segmented image collection is downsampled, and the dynamic part image is extracted to obtain the motion capture image collection; the key frames in the image data in the segmented image collection are screened, and the static part image is extracted to generate the detail recognition image collection.

[0039] Specifically, the image data in the segmented image collection is downsampled. Downsampling is a technique for reducing data size, typically achieved by reducing image resolution or the number of frames. The goal of downsampling is to extract dynamic portions of the video, i.e., portions that change, such as a person's movements. By reducing the image resolution or the number of frames, the dynamic portions are extracted, resulting in a motion capture image collection containing key frames or key images of the person's movements and behaviors in the video. Key frames are then filtered from the image data in the segmented image collection. Key frames are representative frames in the video that contain important information. They typically occur at moments of scene change or significant transitions in action, reflecting major changes or important events in the video content. Key frames are automatically detected using a video analysis algorithm or filtered based on preset rules (such as selecting one frame every certain number of frames). Within the key frames, static regions within the image are analyzed and identified, containing important details. Static portions of the image refer to portions of the video that do not change or change only slightly, such as backgrounds and objects. Static regions are extracted from the key frames to generate a collection of detail recognition images, containing key information of static scenes or objects in the video, such as faces, logos, and specific objects. Through downsampling and dynamic part extraction, dynamic information in the video, such as human movements, can be effectively captured. Through keyframe screening and static part extraction, static information in the video, such as background and objects, can be accurately identified.

[0040] Furthermore, the monitoring network construction module 13 in the live content intelligent monitoring and management system based on deep learning is also used to:

[0041] Video data, audio data, and text data containing normal and illegal content are collected respectively, and normal and illegal content are marked to obtain a video surveillance training data set, an audio surveillance training data set, and a text surveillance training data set, wherein the video surveillance training data set includes a dynamic capture image training set and a detail recognition image training set; supervised training is performed based on the video surveillance training data set, audio surveillance training data set, and text surveillance training data set to obtain a video surveillance channel, an audio surveillance channel, and a text surveillance channel; the video surveillance channel, audio surveillance channel, and text surveillance channel are connected in parallel to obtain the intelligent surveillance network.

[0042] Specifically, video, audio, and text data containing both normal and illegal content, including inappropriate language and threats, are collected from various sources, such as live streaming platforms and social media. The video clips, audio data, and text content are annotated to distinguish between normal and illegal content. The annotated content is divided into video surveillance training datasets, audio surveillance training datasets, and text surveillance training datasets based on data type. The video surveillance training dataset contains video data and is divided into a motion capture image training set and a detail recognition image training set. The motion capture image training set contains images that depict human motion, while the detail recognition image training set contains images that depict scene details. The audio surveillance training dataset contains audio data, used to train models to identify illegal audio content. The text surveillance training dataset contains text data, such as chat logs and comments, used to train models to identify illegal text content. Each dataset contains samples of both normal and illegal content. Using these constructed training datasets, combined with machine learning models such as convolutional neural networks (CNNs) and natural language processing (NLP) models, the video surveillance channel, audio surveillance channel, and text surveillance channel are trained separately. The video surveillance training dataset includes a motion capture image training set (for action recognition) and a detail recognition image training set (for scene recognition). Ensemble learning or multimodal data processing techniques ensure that information from different channels complements and integrates with each other. Trained video, audio, and text monitoring channels are connected in parallel to form an intelligent monitoring network capable of simultaneously processing and analyzing video, audio, and text content to detect and identify illegal content. By connecting these three channels in parallel, efficient monitoring of different content types is achieved, improving both accuracy and efficiency.

[0043] Furthermore, the monitoring network construction module 13 in the live content intelligent monitoring and management system based on deep learning is also used to:

[0044] Based on the dynamic capture image training set and the detail recognition image training set, convolutional neural networks are used for training respectively to obtain motion capture sub-channels and detail recognition sub-channels to constitute the video monitoring channel; the audio monitoring training data set is used in combination with convolutional neural networks for training to obtain the audio monitoring channel; and the text monitoring training data set is trained using a natural language processing model to obtain the text monitoring channel.

[0045] Specifically, image data is extracted from the motion capture image training set and preprocessed, such as normalization, cropping, and rotation, to enhance the model's generalization capabilities. A convolutional neural network architecture suitable for action recognition, such as a CNN, CNN-RNN hybrid model, or other advanced architectures, is selected to identify and capture human motion in videos. The convolutional neural network is trained using the preprocessed image data, and network weights are adjusted using optimization algorithms such as backpropagation and gradient descent. Techniques such as cross-validation are applied to prevent overfitting. After training, the model performance is evaluated on a validation set and deployed as a motion capture subchannel within the video surveillance pipeline. Image data is extracted from the detail recognition image training set and preprocessed, including denoising and enhancement. A convolutional neural network architecture suitable for detail recognition, such as U-Net or SegNet, is selected to identify and capture scene details in videos. The convolutional neural network is trained using the preprocessed image data, and techniques such as transfer learning are applied to improve training efficiency and model performance. After training, the model performance is evaluated on a validation set and deployed as a detail recognition subchannel within the video surveillance pipeline.

[0046] Convert the audio samples in the audio surveillance training dataset into a format suitable for training, such as spectrograms or mel-spectrograms. Use the converted audio data to train a convolutional neural network to identify and classify specific sounds, intonations, or anomalies in the audio signal, thereby generating an audio surveillance channel. Preprocess the text surveillance training dataset, including word segmentation and stop word removal. Select a natural language processing (NLP) model suitable for text classification, such as a recurrent neural network (RNN), long short-term memory (LSTM), or transformer. Use the natural language processing (NLP) model to train the text surveillance training dataset to identify and classify specific words, phrases, or patterns in the text to detect violations or inappropriate content. Through model training, a multifunctional intelligent surveillance network is constructed, including a motion capture sub-channel, a detail recognition sub-channel, an audio surveillance channel, and a text surveillance channel. Each channel is optimized for its specific type of data.

[0047] Furthermore, the violation determination module 16 in the live content intelligent monitoring and management system based on deep learning is also used to:

[0048] Obtain historical violation record information for cluster analysis to generate a violation level comparison table; based on the violation level comparison table, match the video analysis results, audio analysis results, and text analysis results with corresponding violation levels; and perform violation determination and alarm processing based on the violation levels.

[0049] Specifically, historical violation records are collected, including video, audio, and text data, including the type of violation, frequency, and scope of impact. The collected data is cleaned to remove noise and irrelevant information. Clustering algorithms (such as K-Means and DBSCAN) are used to cluster the violation records. Cluster analysis, an unsupervised learning method, aims to group similar violations for better understanding and classification, ensuring high similarity between data points within the same group and low similarity between data points in different groups. Through cluster analysis, violation records are classified into different levels, generating a violation level comparison table. This comparison table should include violation levels corresponding to different violations, such as minor, general, and severe. Based on the violation level comparison table, the video, audio, and text analysis results are matched to corresponding violation levels. The violation content detected in each analysis result is then matched against the rules in the comparison table to determine the severity of the violation. Based on the matched violation level, a violation is determined and appropriate measures are taken, such as recording the violation and sending an alert notification. When a violation is detected, an alert is triggered and sent via various means, such as email, SMS, and instant messaging. The alert should include detailed information about the violation, such as the violation level, time and location of occurrence, and the outcome of the violation handling. By determining the violation level, we can quickly identify and classify violating content, enabling timely response and handling, thereby improving the efficiency and security of content monitoring.

[0050] In summary, the live content intelligent monitoring and management system based on deep learning provided by this application has the following technical effects:

[0051] The data acquisition module is used to collect live content information in real time to obtain a content information collection, wherein the content information collection includes a video information collection, an audio information collection, and a text information collection; the video processing module is used to extract and process the video information collection to obtain a motion capture image collection and a detail recognition image collection; the monitoring network construction module is used to build an intelligent monitoring network, wherein the intelligent monitoring network includes a video monitoring channel, an audio monitoring channel, and a text monitoring channel, wherein the video monitoring channel includes a motion capture sub-channel and a detail recognition sub-channel; the video analysis module is used to input the motion capture image collection and the detail recognition image collection into the motion capture sub-channel and the detail recognition sub-channel of the intelligent monitoring network respectively to obtain video analysis results; the text analysis module is used to input the audio information collection and the text information collection into the audio monitoring channel and the text monitoring channel of the intelligent monitoring network respectively to obtain audio analysis results and text analysis results; the violation determination module is used to perform violation determination and alarm processing based on the video analysis results, audio analysis results, and text analysis results. In other words, by building a multi-channel parallel monitoring and review mechanism, all-round and real-time monitoring of live broadcast content is achieved, achieving the technical effect of improving the efficiency and accuracy of content review.

[0052] In the second embodiment, based on the same inventive concept as the live content intelligent monitoring and management system based on deep learning in the previous embodiment, this application also provides a live content intelligent monitoring and management method based on deep learning, please refer to the attached Figure 2 , the live content intelligent monitoring and management method based on deep learning includes:

[0053] Live content information is collected in real time to obtain a content information collection, wherein the content information collection includes a video information collection, an audio information collection, and a text information collection; the video information collection is extracted and processed to obtain a motion capture image collection and a detail recognition image collection; an intelligent monitoring network is constructed, wherein the intelligent monitoring network includes a video monitoring channel, an audio monitoring channel, and a text monitoring channel, wherein the video monitoring channel includes a motion capture sub-channel and a detail recognition sub-channel; the motion capture image collection and the detail recognition image collection are respectively input into the motion capture sub-channel and the detail recognition sub-channel of the intelligent monitoring network to obtain a video analysis result; the audio information collection and the text information collection are respectively input into the audio monitoring channel and the text monitoring channel of the intelligent monitoring network to obtain an audio analysis result and a text analysis result; based on the video analysis result, the audio analysis result, and the text analysis result, violation determination and alarm processing are respectively performed.

[0054] Furthermore, the extracting and processing the video information collection to obtain the motion capture image collection and the detail recognition image collection includes:

[0055] Constructing a semantic segmentation model; inputting the video information collection into the semantic segmentation model for semantic segmentation to obtain a segmented image collection; and processing based on the segmented image collection to obtain a motion capture image collection and a detail recognition image collection.

[0056] Furthermore, the constructing of the semantic segmentation model includes:

[0057] Based on big data, multiple daily life images are collected to obtain multiple sample image information sets; based on the differences between human body features and background features, the image information in the multiple sample image information sets is segmented to obtain multiple sample image segmentation result sets; the multiple sample image information sets and the multiple sample image segmentation result sets are divided to obtain training data sets and verification data sets; based on the training data sets and verification data sets, a convolutional neural network model is supervised trained, verified and tested to obtain the semantic segmentation model.

[0058] Furthermore, the processing based on the segmented image collection to obtain the motion capture image collection and the detail recognition image collection includes:

[0059] The image data in the segmented image collection is downsampled, and the dynamic part image is extracted to obtain the motion capture image collection; the key frames in the image data in the segmented image collection are screened, and the static part image is extracted to generate the detail recognition image collection.

[0060] Furthermore, the intelligent monitoring network is constructed, and the intelligent monitoring network includes a video monitoring channel, an audio monitoring channel, and a text monitoring channel, including:

[0061] Video data, audio data, and text data containing normal and illegal content are collected respectively, and normal and illegal content are marked to obtain a video surveillance training data set, an audio surveillance training data set, and a text surveillance training data set, wherein the video surveillance training data set includes a dynamic capture image training set and a detail recognition image training set; supervised training is performed based on the video surveillance training data set, audio surveillance training data set, and text surveillance training data set to obtain a video surveillance channel, an audio surveillance channel, and a text surveillance channel; the video surveillance channel, audio surveillance channel, and text surveillance channel are connected in parallel to obtain the intelligent surveillance network.

[0062] Furthermore, the supervised training is performed based on the video surveillance training data set, the audio surveillance training data set, and the text surveillance training data set to obtain the video surveillance channel, the audio surveillance channel, and the text surveillance channel, respectively, including:

[0063] Based on the dynamic capture image training set and the detail recognition image training set, convolutional neural networks are used for training respectively to obtain motion capture sub-channels and detail recognition sub-channels to constitute the video monitoring channel; the audio monitoring training data set is used in combination with convolutional neural networks for training to obtain the audio monitoring channel; and the text monitoring training data set is trained using a natural language processing model to obtain the text monitoring channel.

[0064] Furthermore, the violation determination and alarm processing are performed based on the video analysis results, audio analysis results, and text analysis results, respectively, including:

[0065] Obtain historical violation record information for cluster analysis to generate a violation level comparison table; based on the violation level comparison table, match the video analysis results, audio analysis results, and text analysis results with corresponding violation levels; and perform violation determination and alarm processing based on the violation levels.

[0066] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. Figure 1 The live content intelligent monitoring and management system based on deep learning and the specific examples in Example 1 are also applicable to the live content intelligent monitoring and management method based on deep learning in this embodiment. Through the above detailed description of the live content intelligent monitoring and management system based on deep learning, those skilled in the art can clearly understand the live content intelligent monitoring and management method based on deep learning in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here. As for the method disclosed in the embodiment, since it corresponds to the system disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the system part description.

[0067] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

[0068] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application is intended to include these modifications and variations.

Claims

1. Live content intelligent monitoring and management system based on deep learning, characterized by: include: A data collection module, the data collection module is used to collect live content information in real time to obtain a content information collection, wherein the content information collection includes a video information collection, an audio information collection and a text information collection; A video processing module, the video processing module is used to extract and process the video information collection to obtain a motion capture image collection and a detail recognition image collection; A monitoring network construction module, wherein the monitoring network construction module is used to construct an intelligent monitoring network, wherein the intelligent monitoring network includes a video monitoring channel, an audio monitoring channel, and a text monitoring channel, wherein the video monitoring channel includes a motion capture sub-channel and a detail recognition sub-channel; A video analysis module, wherein the video analysis module is used to input the motion capture image collection and the detail recognition image collection into the motion capture sub-channel and the detail recognition sub-channel of the intelligent monitoring network respectively to obtain a video analysis result; A text analysis module, wherein the text analysis module is used to input the audio information collection and the text information collection into the audio monitoring channel and the text monitoring channel of the intelligent monitoring network respectively, to obtain an audio analysis result and a text analysis result; A violation determination module is used to perform violation determination and alarm processing based on the video analysis results, audio analysis results and text analysis results.

2. The live content intelligent monitoring and management system based on deep learning as claimed in claim 1, characterized in that: The video processing module is also used for: Build a semantic segmentation model; Inputting the video information collection into the semantic segmentation model to perform semantic segmentation to obtain a segmented image collection; Processing is performed based on the segmented image collection to obtain a motion capture image collection and a detail recognition image collection.

3. The live content intelligent monitoring and management system based on deep learning as claimed in claim 2, characterized in that: The video processing module is also used for: Based on big data, multiple daily life images are collected to obtain multiple sample image information sets; According to the difference between human body features and background features, image information in the plurality of sample image information sets is segmented to obtain a plurality of sample image segmentation result sets; Dividing the plurality of sample image information sets and the plurality of sample image segmentation result sets to obtain a training data set and a verification data set; The convolutional neural network model is supervisedly trained, verified and tested based on the training data set and the verification data set to obtain the semantic segmentation model.

4. The live content intelligent monitoring and management system based on deep learning as claimed in claim 2, characterized in that: The video processing module is also used for: Down-sampling the image data in the segmented image collection, extracting the dynamic part image, and obtaining the motion capture image collection; The key frames in the image data in the segmented image collection are screened, static partial images are extracted, and the detail recognition image collection is generated.

5. The live content intelligent monitoring and management system based on deep learning as claimed in claim 1, characterized in that: The monitoring network building module is also used for: Collect video data, audio data and text data containing normal and illegal content respectively, and mark them as normal and illegal, so as to obtain a video surveillance training data set, an audio surveillance training data set and a text surveillance training data set, wherein the video surveillance training data set includes a dynamic capture image training set and a detail recognition image training set; Based on the video surveillance training data set, the audio surveillance training data set and the text surveillance training data set, supervised training is performed respectively to obtain a video surveillance channel, an audio surveillance channel and a text surveillance channel; The video monitoring channel, the audio monitoring channel and the text monitoring channel are connected in parallel to obtain the intelligent monitoring network.

6. The live content intelligent monitoring and management system based on deep learning as claimed in claim 5, characterized in that: The monitoring network building module is also used for: Based on the dynamic capture image training set and the detail recognition image training set, a convolutional neural network is used to perform training respectively to obtain a motion capture sub-channel and a detail recognition sub-channel to constitute the video monitoring channel; Using the audio monitoring training data set in combination with a convolutional neural network for training, to obtain the audio monitoring channel; The text monitoring training data set is trained using a natural language processing model to obtain the text monitoring channel.

7. The live content intelligent monitoring and management system based on deep learning as claimed in claim 1, characterized in that: The violation determination module is also used for: Obtain historical violation record information for cluster analysis and generate a violation level comparison table; Based on the violation level comparison table, matching corresponding violation levels for the video analysis results, the audio analysis results, and the text analysis results; Based on the violation level, violation determination and alarm processing are performed.

8. A live broadcast content intelligent monitoring and management method based on deep learning, characterized in that: The method is performed by the live content intelligent monitoring and management system based on deep learning according to any one of claims 1 to 7, wherein the live content intelligent monitoring and management method based on deep learning comprises: Collecting live broadcast content information in real time to obtain a content information collection, wherein the content information collection includes a video information collection, an audio information collection, and a text information collection; Extracting and processing the video information collection to obtain a motion capture image collection and a detail recognition image collection; Building an intelligent monitoring network, the intelligent monitoring network includes a video monitoring channel, an audio monitoring channel and a text monitoring channel, wherein the video monitoring channel includes a motion capture sub-channel and a detail recognition sub-channel; Inputting the motion capture image collection and the detail recognition image collection into the motion capture sub-channel and detail recognition sub-channel of the intelligent monitoring network respectively to obtain a video analysis result; Input the audio information collection and the text information collection into the audio monitoring channel and the text monitoring channel of the intelligent monitoring network respectively to obtain audio analysis results and text analysis results; Based on the video analysis results, audio analysis results and text analysis results, violation determination and alarm processing are performed respectively.

Images