Internet live broadcast illegal behavior monitoring method and system based on image and voice recognition

Abstract

The invention discloses a method and a system for monitoring internet live broadcast illegal behaviors based on image and voice recognition, and relates to the technical field of live broadcast monitoring. The method comprises the following steps: acquiring live stream data of each anchor and verifying data integrity; processing the live broadcast stream data to obtain preprocessed live broadcast stream data; performing feature recognition on the video frame set by using the determined image violation processing model, and outputting image violation marking information; performing text processing on the voice-to-text data by using the determined text processing model, and outputting text violation marking information; based on the image violation mark information and the text violation mark information, determining whether the anchor violates the rule or not, and giving an alarm when the anchor violates the rule; and the data acquisition parameters are corrected based on the historical violation data of the anchor, so that the problem that it is difficult to accurately acquire and process the live broadcast data to identify Internet live broadcast violation behaviors and optimize the monitoring parameters according to the historical violation data by using image and voice recognition technologies at the same time in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of live broadcast monitoring, and specifically provides a method and system for monitoring Internet live broadcast violations through image and voice recognition. Background Art

[0002] With the development of Internet technology, live broadcasts in various scenarios have developed rapidly. However, there are also various chaos in the current live broadcast process on each live broadcast platform. For example, the smoking behavior in the live broadcast scenario, that is, the anchor smokes in the live broadcast scenario. Smoking is not only harmful to health, but also brings bad guidance and is not conducive to the healthy growth of teenagers. The Chinese utility model patent with the publication number CN113705370A discloses a method, device, electronic device and storage medium for detecting violations in a live broadcast room, which relates to the technical field of Internet mobile applications. After extracting frames from the video to be detected to obtain a number of video frames, the number of video frames are then sequentially input into a violation detection model to determine whether there are violations in the video frames and mark the violations as abnormal. If it is determined that the first video frame has a first violation according to the abnormal mark of the first video frame, it is confirmed whether the second video frame has a second violation according to the abnormal mark of the second video frame, where the second video frame is the previous video frame of the first video frame among the number of video frames. Finally, after it is determined that the second video frame has a second violation, it is determined that the video to be detected has a violation in the live broadcast room according to the first violation and the second violation. This detection method ensures the accuracy of detecting violations in the live broadcast room through a multi-frame fusion mechanism.

[0003] However, the existing technology has the problem that it is difficult to simultaneously use image and voice recognition technologies to accurately obtain and process live broadcast data to identify Internet live broadcast violations and optimize monitoring parameters according to historical violation data. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology, the present invention provides a method and system for monitoring Internet live broadcast violations through image and voice recognition, which solves the problem that the existing technology is difficult to simultaneously use image and voice recognition technologies to accurately obtain and process live broadcast data to identify Internet live broadcast violations and optimize monitoring parameters according to historical violation data.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A monitoring method for Internet live broadcast violation behaviors of image and voice recognition, comprising the following steps: obtaining live stream data of each anchor from a live broadcast platform based on determined data collection parameters and verifying the data integrity, where the live stream data includes original video stream data and original audio stream data; processing the live stream data through a determined data processing mode to obtain preprocessed live stream data, including a video frame set and speech-to-text data; using a determined image violation processing model to perform feature recognition on the video frame set, and outputting image violation marking information, including the violation type, violation probability, and image violation timestamp of each image frame; using a determined text processing model to perform text processing on the speech-to-text data, and outputting text violation marking information, including the violated text paragraph and text violation timestamp; determining whether the anchor violates the regulations based on the image violation marking information and the text violation marking information and issuing an alarm when a violation occurs; and correcting the data collection parameters based on the historical violation data of the anchor.

[0006] Further, verifying the data integrity of the live stream data of each anchor includes the following steps: obtaining the data integrity coefficient I of each anchor comp and the data format correctness index I form : where D act is the actual returned data volume, D exp is the data volume expected to be obtained by the request, N corr is the number of fields with correct formats in the data, and N total is the total number of fields in the data; if the data integrity coefficient is greater than the data integrity threshold corresponding to the determined anchor and the data format correctness index is greater than the data format correctness threshold corresponding to the determined anchor, the data integrity of the anchor is normal, otherwise, the data integrity is abnormal, and the live stream data of the anchor is obtained again; where the data collection parameters include the API application frequency and the data acquisition request interval time of the network crawler program; the data collection parameters, the data integrity threshold, the data format correctness threshold, the data processing mode, the image violation processing model, and the text processing model are all determined by obtaining the influencing factors corresponding to each anchor.

[0007] Further, the determination method of the data collection parameters, the data integrity threshold, and the data format correctness threshold is as follows: obtaining the set influencing factor - data processing parameter mapping set stored in the database, comparing the influencing factor with each set influencing factor one by one, calculating the similarity based on the Euclidean distance, determining the set influencing factor corresponding to the minimum similarity, and obtaining the corresponding data processing parameters from the database, including the data collection parameters, the data integrity threshold, and the data format correctness threshold;

[0008] The data processing modes include a video stream processing mode and an audio stream processing mode. The video stream processing mode includes an adaptive histogram equalization processing mode and a Gaussian filtering denoising processing mode. The audio stream processing mode includes a first-level text processing mode and a second-level text processing mode. The image violation processing models include a Mask R-CNN model and a Faster R-CNN model. The text processing models include a text processing model based on the Transformer architecture and a keyword matching processing model. The determination methods for the data processing mode, the image violation processing model, and the text processing model are as follows:

[0009] Compare the influence factors corresponding to each anchor with the influence threshold. If the influence factor is greater than the influence threshold, the video stream processing mode is the adaptive histogram equalization processing mode, the audio stream processing mode is the first-level text processing mode, the image violation processing model is the Mask R-CNN model, and the text processing model is the text processing model based on the Transformer architecture. If the influence factor is not greater than the influence threshold, the video stream processing mode is the Gaussian filtering denoising processing mode, the audio stream processing mode is the second-level text processing mode, the image violation processing model is the Faster R-CNN model, and the text processing model is the keyword matching processing model.

[0010] Furthermore, the method for obtaining the influence factor is as follows: Obtain the live streaming platform influence data and the anchor influence data corresponding to each anchor, and determine the live streaming platform influence coefficient λ1 and the anchor influence coefficients of each anchor. i is the anchor number; Based on the live streaming platform influence coefficient λ1 and the anchor influence coefficients of each anchor. Obtain the initial influence factor. Determine the correction factor based on the real-time live streaming data of each anchor. For the initial influence factor. Perform real-time correction to obtain the influence factor. e is the natural constant.

[0011] Furthermore, the method for obtaining the live streaming platform influence coefficient λ1 is as follows:

[0012] λ1 = λ Sc + λ PF + λ B + λ TH + λ QL ;

[0013] Among them, λ Sc is the market share coefficient, obtained based on a logistic regression model, and λ PF is the user word-of-mouth coefficient, obtained based on a calculation model of the average score and the standard deviation of the scores, and λ Bis the media exposure coefficient, determined based on the analytic hierarchy process, λ TH is the technology input-output ratio coefficient, determined based on the technology innovation funds and the effect of achievement transformation, λ QL is the user growth potential coefficient, determined based on the historical user growth rate;

[0014] Host influence coefficient The acquisition method is as follows:

[0015]

[0016] Among them, is the host fan scale factor, is the host live activity factor, is the host content quality factor, and X is the total number of hosts in the live streaming platform.

[0017] Furthermore, for the correction factor, the acquisition method is as follows: Obtain the real-time data during the host's live broadcast, including the growth rate of the number of viewers xZ per unit time during the live broadcast, the keyword density ρ of the audience per unit time during the live broadcast gc and the total number of online people P in the current unit time during the live broadcast zx , and obtain the correction factor based on the correction factor calculation model Among them, is the average value of the number of online people during the host's historical live broadcasts.

[0018] Furthermore, determine whether the host violates the regulations based on the image violation marking information and the text violation marking information, and give an alarm when there is a violation, including the following steps: If the image violation timestamp is the same as the text violation timestamp, and the violation probability is greater than the set violation threshold and there are violation keywords in the violated text paragraph, it is determined that a violation has occurred, and an alarm message is sent to the operator of the live streaming platform; If the image violation timestamp is the same as the text violation timestamp, and the violation probability is greater than the set violation threshold or there are violation keywords in the violated text paragraph, it is determined that a violation has occurred, and an alarm message is sent to the operator of the live streaming platform.

[0019] Furthermore, correct the data collection parameters based on the host's historical violation data, including the following steps: Construct a full-platform risk baseline matrix based on the historical violation data of each host, calculate the matrix elements, where the row dimension of the risk baseline matrix represents the violation risk base number of the violation type, and the column dimension represents the time window; Obtain the real-time violation feature vector of the live streaming platform within the monitoring time window, and calculate the dynamic violation risk coefficient by combining the platform risk baseline matrix and the host influence coefficient of each host; Determine the platform violation risk level based on the risk level mapping relationship, and match the corrected data collection parameters stored in the database based on the platform violation risk level.

[0020] Furthermore, the calculation method of matrix elements is as follows:

[0021]

[0022] where w i′j′ is the matrix element in the full-platform risk baseline matrix, δ(C i′，j′ ) is the number of violations of the i'-th violation type at time window j', is the maximum number of violations of the i'-th violation type, i' is the number of the violation type, j' is the number of the time window, ξ is the time decay factor, t now is the current time, t wg is the time of the most recent violation, and J is the total number of violation types.

[0023] The method for obtaining the dynamic violation risk coefficient is as follows:

[0024]

[0025] where λ risk is the dynamic violation risk coefficient, F eral is the real-time violation feature vector, is the transpose matrix of the full-platform risk baseline matrix M base , is the mean value of the influence factor.

[0026] Compare the dynamic violation risk coefficient with each risk threshold interval stored in the database to determine the risk threshold interval to which the dynamic violation risk coefficient belongs. Then, according to the mapping relationship between the risk threshold interval and the risk level, determine the platform violation risk level. After that, obtain the corresponding stored corrected data acquisition parameters from the database according to the platform violation risk level.

[0027] A monitoring system for Internet live broadcast violation behaviors of image and voice recognition, which is used for the monitoring method of Internet live broadcast violation behaviors of image and voice recognition described above, includes: a live stream data acquisition module, which is used to obtain the live stream data of each anchor from the live broadcast platform based on the determined data acquisition parameters and verify the data integrity. The live stream data includes original video stream data and original audio stream data; a data preprocessing module, which is used to process the live stream data through the determined data processing mode to obtain preprocessed live stream data, including a video frame set and speech-to-text data; an image processing module, which is used to perform feature recognition on the video frame set by using the determined image violation processing model and output image violation marking information, including the violation type, violation probability, and image violation timestamp of each image frame; a text processing module, which is used to perform text processing on the speech-to-text data by using the determined text processing model and output text violation marking information, including the violation text paragraph and text violation timestamp; a violation judgment module, which is used to determine whether the anchor violates the rules based on the image violation marking information and the text violation marking information and give an alarm when a violation occurs; a collection parameter correction module, which is used to correct the data collection parameters based on the historical violation data of the anchor.

[0028] The present invention has the following beneficial effects:

[0029] The monitoring method and system for Internet live broadcast violation behaviors of image and voice recognition obtain live stream data by determining data acquisition parameters and verify the integrity, ensuring the availability of the data; obtain a video frame set and speech-to-text data through a specific data processing mode, providing a basis for subsequent analysis; use image and text processing models to identify violation information respectively, comprehensively capturing violation clues; determine whether the anchor violates the rules based on this information and give an alarm, promptly stopping the violation behavior; can also correct the data collection parameters according to the historical violation data of the anchor, realizing dynamic optimization of monitoring, effectively maintaining the order of the live broadcast platform, and ensuring a good experience for users.

[0030] Of course, it is not necessary for any product implementing the present invention to simultaneously achieve all the above-mentioned advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of the monitoring method for Internet live broadcast violation behaviors of image and voice recognition of the present invention.

[0032] Figure 2 It is a block diagram of the monitoring system for Internet live broadcast violation behaviors of image and voice recognition of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0033] Please refer to Figure 1, an embodiment of the present invention provides a technical solution: a method for monitoring Internet live broadcast violations of image and voice recognition, including the following steps: obtaining the live stream data of each anchor from the live broadcast platform based on the determined data collection parameters and verifying the data integrity, where the live stream data includes the original video stream data and the original audio stream data;

[0034] Obtain the data integrity coefficient I of each anchor comp And the data format correctness index I form :

[0035]

[0036] Wherein, D act Is the actual returned data volume, D exp Is the data volume expected to be obtained by the request, N corr Is the number of fields with correct format in the data, N total Is the total number of fields in the data; these two indicators evaluate the data quality in a quantitative manner from two key dimensions of data volume and data format, providing an accurate basis for subsequent judgment of whether the data meets the monitoring requirements, making the data quality evaluation more scientific and objective.

[0037] If the data integrity coefficient is greater than the determined data integrity threshold of the corresponding anchor and the data format correctness index is greater than the determined data format correctness threshold of the corresponding anchor, then the data integrity of the anchor is normal, otherwise, the data integrity is abnormal, and the live stream data of the anchor is obtained again; it can effectively ensure that the live stream data used for subsequent violation behavior analysis is complete and in the correct format, avoiding missed or misjudged violations due to data loss or format errors, thus ensuring the accuracy of the monitoring results.

[0038] Wherein, the data collection parameters include the API application frequency and the data acquisition request interval time of the network crawler program; the data collection parameters, the data integrity threshold, the data format correctness threshold, the data processing mode, the image violation processing model, and the text processing model are all determined by obtaining the corresponding influence factors of each anchor. It is clear that the data collection parameters (such as the API application frequency and the data acquisition request interval time of the network crawler program), the data integrity threshold, the data format correctness threshold, the data processing mode, the image violation processing model, and the text processing model are all determined by obtaining the corresponding influence factors of each anchor. This enables the monitoring method to dynamically adjust the relevant parameters and modes of data collection and processing according to the specific situations of different live broadcast platforms and anchors, improving the pertinence and adaptability of the monitoring system, and thus more effectively identifying and processing various live broadcast violations.

[0039] The determination methods of the data collection parameters, the data integrity threshold, and the data format correctness threshold are as follows:

[0040] Obtain the set of mapping between the set impact factors and data processing parameters stored in the database. Compare each impact factor with the set impact factors one by one, calculate the similarity based on the Euclidean distance, determine the set impact factor corresponding to the minimum similarity, and obtain the corresponding data processing parameters from the database, including data acquisition parameters, data integrity threshold, and data format correctness threshold. This method enables the data acquisition link to be accurately set according to the comprehensive situation of the live streaming platform and the anchor. For example, for anchors with different influences, reasonably adjust the API application frequency and the data acquisition request interval of the web crawler program, which not only ensures the efficiency of data acquisition, but also avoids overloading the live streaming platform, and at the same time ensures that the quality of the acquired data meets the monitoring requirements.

[0041] The data processing modes include video stream processing mode and audio stream processing mode. Among them, the video stream processing mode includes adaptive histogram equalization processing mode and Gaussian filter denoising processing mode, and the audio stream processing mode includes the first-level text processing mode and the second-level text processing mode. The image violation processing models include Mask R-CNN model and Faster R-CNN model, and the text processing models include Transformer architecture text processing model and keyword matching processing model. The determination methods of the data processing mode, image violation processing model, and text processing model are as follows:

[0042] Compare the impact factors corresponding to each anchor with the impact threshold.

[0043] If the impact factor is greater than the impact threshold, the video stream processing mode is the adaptive histogram equalization processing mode, the audio stream processing mode is the first-level text processing mode, the image violation processing model is the Mask R-CNN model, and the text processing model is the Transformer architecture text processing model.

[0044] First-level text processing mode: First, perform format conversion to convert it into a standard format supported by the speech recognition engine, such as the common PCM (Pulse-Code Modulation) format. In the preprocessing stage, use a noise reduction algorithm to remove background noise from the audio, and select a more powerful and accurate speech recognition engine, such as the enhanced version of Baidu Speech Recognition based on deep learning. This engine has a more complex neural network architecture and can better process high-complexity audio data. To further improve the recognition accuracy, increase the amount of training data for the language model. Select new training data from a large amount of text data related to the live content, including live scripts, professional literature in related fields, discussions about the live theme on social media, etc., to enrich the language model's learning ability for language expressions in the live scenario. Input the preprocessed audio signal into the speech recognition engine with adjusted parameters for recognition. The engine outputs the recognized result text, which is the speech-to-text content corresponding to the high-impact audio stream and is used for subsequent text keyword matching and semantic analysis.

[0045] The Mask R-CNN model is developed based on Faster R-CNN and can achieve object detection and instance segmentation simultaneously. When initializing the model, load the pre-trained weight parameters and set the set of pre-trained weight parameters. At the same time, according to the characteristics of the live image, such as size, number of color channels, etc., set the input layer parameters of the model. Assume the size of the live image is H×W×C, where H is the image height (pixels), W is the image width (pixels), and C is the number of color channels (e.g., for an RGB image, C = 3), and adjust the input layer of the model to a structure adapted to this size. Input the image frames of the high-impact platform and the host after image frame extraction and preprocessing (such as adaptive histogram equalization processing) into the Mask R-CNN model. Before input, perform normalization processing on the image frames, mapping the image pixel values from the range of 0 to 255 to the range of 0 - 1. Normalization processing helps to accelerate model training and improve the stability of the model.

[0046] The Mask R-CNN model performs a forward propagation operation on the input normalized image frame. First, the model extracts deep features of the image through a backbone network (such as ResNet or Feature Pyramid Network, FPN). During the feature extraction process, convolutional operations in different layers perform convolution operations on the feature map through convolutional kernels. Then, a Region Proposal Network (RPN) generates candidate regions that may contain objects. Next, a Region of Interest (RoI) Align operation is performed on each candidate region on the feature map to obtain a fixed-size feature vector for subsequent classification and segmentation predictions. The Mask R-CNN model performs classification and segmentation predictions on the feature vector of each candidate region to determine whether it is a feature of a violation image. At the same time, for the detected violation regions, instance segmentation is performed to obtain a segmentation mask, marking the exact contour of the violation object. For the detected violation image frame, the corresponding timestamp is recorded and the violation image frame marking information is stored.

[0047] Preprocess the speech-to-text content of high-impact categories. Convert the text to lowercase, removing irrelevant information such as punctuation marks and special characters. Taking the removal of punctuation marks as an example, by defining a set of punctuation marks and using string replacement operations, for each character in the text, if it belongs to the set of punctuation marks, it is replaced with an empty character to obtain the preprocessed text. Select a semantic analysis model based on the Transformer architecture, such as the BERT (Bidirectional Encoder Representations from Transformers) model. Load the pre-trained BERT model parameters, which are trained on a large-scale text corpus and contain rich language semantic knowledge. According to the characteristics of the live content, configure the input parameters of the model. For example, truncate or pad the text to a fixed length (such as 128 words). Based on the semantic representation output by the model, perform violation semantic mining through a fully connected layer and a classifier. Output the output probability of the classifier. If the output result is greater than the set text violation threshold, it is determined that the text has violation semantics, and the corresponding text paragraph and timestamp are marked.

[0048] If the impact factor is not greater than the impact threshold, the video stream processing mode is the Gaussian filtering denoising processing mode, the audio stream processing mode is the second-level text processing mode, the image violation processing model is the Faster R-CNN model, and the text processing model is the keyword matching processing model.

[0049] Second level text processing mode: The format is also converted to PCM format, and a simple noise reduction algorithm is used, such as a noise reduction method based on mean filtering. A basic configuration speech recognition engine is used, such as the open source Kaldi speech recognition engine basic version. This engine has low computing resource requirements and is suitable for processing audio streams that require relatively low recognition accuracy. The preprocessed audio signal is input into the basic configuration speech recognition engine for recognition. The engine outputs the recognition result text, which is the speech-to-text content corresponding to other live streams, and is used for subsequent text keyword matching and semantic analysis.

[0050] If the Faster R-CNN model is selected, the initialization process is similar to that of the Mask R-CNN model. The pre-trained weight parameters are loaded, and the input layer parameters are set according to the live image size. The low-impact live stream image frames after image frame extraction and preprocessing (such as Gaussian filtering) are input into the selected model. The image frames are also normalized to obtain normalized image frames. The normalization formula is consistent with the high-impact image frame processing. The Faster R-CNN model operates on the input normalized image frames. The backbone network extracts deep image features to obtain the feature map region proposal network (RPN) to generate a set of candidate regions on the feature map. Then, the candidate regions are RoIPool ing (similar to RoI Align, but slightly different in calculation method) to obtain a fixed-size feature vector for subsequent classification and bounding box regression. The model classifies and predicts the feature vectors of the candidate regions, records the timestamps corresponding to the image frames where the illegal features are detected, and stores the illegal image frame marking information for subsequent comprehensive judgment with speech-to-text content.

[0051] When the text processing model is a keyword matching processing model, keyword matching is performed on other types of speech-to-text content after preprocessing. Use a string matching algorithm (such as a naive string matching algorithm or a more efficient KMP algorithm) to search for keywords in the keyword library in the text. The keyword library is a predefined set of keywords related to illegal content, such as "violence" and "pornography". The keyword library can be continuously expanded and optimized according to the type of live content and common illegal scenarios.

[0052] Process the live stream data through a determined data processing mode to obtain preprocessed live stream data, including a video frame set and speech-to-text data; based on the comparison result between the impact factor and the impact threshold, select an appropriate video stream processing mode, audio stream processing mode, image violation processing model, and text processing model for different live streamers. For example, high-influence live streamers adopt more complex and precise processing modes and models, such as the adaptive histogram equalization processing mode to improve image quality and assist in image violation recognition, and the Transformer architecture text processing model to mine complex semantic violations; low-influence live streamers adopt relatively simple and efficient modes and models. Such a configuration can make full use of resources, improve the efficiency and accuracy of violation recognition, and reduce the system operation cost while ensuring the monitoring effect.

[0053] The method for obtaining the impact factor is as follows: Obtain the live streaming platform impact data and the live streamer impact data corresponding to each live streamer, and determine the live streaming platform impact coefficient λ1 and the live streamer impact coefficient of each live streamer i is the number of the live streamer; based on the live streaming platform impact coefficient λ1 and the live streamer impact coefficient of each live streamer Obtain the initial impact factor Determine the correction factor based on the real-time live stream data of each live streamer For the initial impact factor Perform real-time correction to obtain the impact factor

[0054] e is the natural constant.

[0055] The method for obtaining the live streaming platform impact coefficient λ1 is as follows: λ1 = λ Sc + λ PF + λ B + λ TH + λ QL ; where λ Sc is the market share coefficient, obtained based on a logistic regression model, λ PF is the user word-of-mouth coefficient, obtained based on a calculation model of the average score and the standard deviation of the score, λ B is the media exposure coefficient, determined based on the analytic hierarchy process, λ TH is the technology input-output ratio coefficient, determined based on the technology innovation funds and the effect of achievement transformation, λ QL is the user growth potential coefficient, determined based on the historical user growth rate;

[0056] where U p is the total number of registered users of the live streaming platform, U t is the total number of users in the live streaming industry, M is the user proportion, and a and b are parameters obtained through training of the logistic regression model.

[0057] λ PF = cR - dσ, where c is the weight factor of the average score R, and d is the weight factor of the standard deviation of the score σ.

[0058] E comp is the comprehensive media exposure index. E max is the maximum value among the comprehensive media exposure indices of all live streaming platforms. Nm is the number of reports by mainstream media (such as TV stations, well-known newspapers, large news websites), Nn is the number of reports by professional media in the industry, Ns is the number of popular topics on social media (such as Weibo, Douyin), and w m is the weight factor of Nm, w n is the weight factor of Nn, w s is the weight factor of Ns.

[0059] λ TH = k*T + m*A, where k is the adjustment factor of the proportion of technology R & D expenses, T is the proportion of technology R & D expenses, I tech is the technology R & D expense, R rev is the total operating income in the same period, and m is the weight factor of the technology input achievement conversion coefficient A. The technology input achievement conversion coefficient A is the weighted sum value of indicators such as the number of user growth and the improvement of user activity after the application of new technologies.

[0060] U p,new is the expected number of new users within the predicted time period T new and τ is the average user growth rate in the historical period.

[0061] The calculation of the live streaming platform influence coefficient synthesizes multiple dimensions such as the market share coefficient, user word-of-mouth coefficient, media exposure coefficient, technology input-output ratio coefficient, and user growth potential coefficient. The market share coefficient is obtained based on the logistic regression model, the user word-of-mouth coefficient is calculated according to the average score and standard deviation, the media exposure coefficient is determined using the analytic hierarchy process, the technology input-output ratio coefficient is obtained by combining the technology innovation funds and the achievement conversion effect, and the user growth potential coefficient is determined according to the historical user growth rate. This multi-dimensional calculation method can comprehensively and accurately evaluate the position of the live streaming platform in the market, user evaluation, industry influence, and development potential, providing reliable platform-level data support for determining the influencing factors in the follow-up.

[0062] Host influence coefficient The acquisition method is as follows: Among them, is the host fan scale factor, is the host live streaming activity factor, is the host content quality factor, and X is the total number of hosts in the live streaming platform.

[0063] U f is the number of fans of the host, and U f,avg is the average number of fans of all hosts on the platform.

[0064] D i,g is the average number of online viewers in the g-th live broadcast of the host, and H i,g is the duration of the g-th live broadcast of the host. G is the number of live broadcasts of the host within the statistical period, and D iz is the average number of online viewers in the z-th live broadcast of the i-th host, and H iz is the duration of the z-th live broadcast of the i-th host.

[0065] V i,g is the content quality score of the video of the host's g-th live broadcast, which is scored by a professional review team from multiple dimensions such as content innovation, professionalism, and entertainment, and R i,g is the audience feedback score of the video of the host's g-th live broadcast, which is obtained by the weighted sum of the number of times of audience likes, comments, shares, etc., and R i,g,min and R i,g,max are the minimum and maximum values of the audience feedback scores of all hosts' live broadcast videos within the statistical period.

[0066] The host influence coefficient is calculated through the host fan scale factor, the host live broadcast activity factor, and the host content quality factor, and is standardized in combination with the total number of hosts on the live broadcast platform. Among them, the fan scale factor reflects the audience base of the host, the live broadcast activity factor reflects the live broadcast frequency of the host and the interaction degree with the audience, and the content quality factor evaluates the quality of the host's live broadcast content from aspects such as professional review and audience feedback. This calculation method can accurately measure the influence of each host within the platform, which helps the monitoring system to more reasonably allocate monitoring resources and select appropriate monitoring strategies according to the actual influence of the host, and improve the accuracy and effectiveness of monitoring the violation behaviors of different hosts.

[0067] The correction factor is obtained as follows: Obtain the real-time data during the host's live broadcast, including the growth rate xZ of the number of viewers per unit time during the live broadcast, the keyword density ρ of the audience per unit time during the live broadcast gc and the total number of online people P at the current unit time during the live broadcast zx , and obtain the correction factor based on the correction factor calculation model Among them, is the average value of the number of online people during the host's historical live broadcast.

[0068] By obtaining the real-time data during the host's live broadcast, such as the growth rate of the number of viewers per unit time, the keyword density of the audience, and the total number of online people in the current unit time, and combining with the average value of the online number during the host's historical live broadcast process, a correction factor is calculated. This method can reflect the popularity changes, audience feedback, and content orientation of the host during the live broadcast in real time. When the growth rate of the number of viewers is high, the keyword density of the audience is abnormal, or the total number of online people fluctuates greatly, the correction factor will be adjusted accordingly, and then the influencing factor will be corrected in real time. This enables the monitoring system to adjust the data acquisition parameters, data processing mode, and violation judgment criteria in a timely manner according to the real-time dynamics of the host's live broadcast, enhancing the timeliness and accuracy of monitoring, and more effectively identifying and dealing with violations during the live broadcast.

[0069] Use the determined image violation processing model to perform feature recognition on the video frame set, and output image violation marking information, including the violation type, violation probability, and image violation timestamp of each image frame;

[0070] Use the determined text processing model to process the speech-to-text data, and output text violation marking information, including the violated text paragraph and the text violation timestamp;

[0071] Based on the image violation marking information and the text violation marking information, determine whether the host is in violation and give an alarm when in violation; if the image violation timestamp and the text violation timestamp are the same, and the violation probability is greater than the set violation threshold and there are violation keywords in the violated text paragraph, it is determined to be in violation, and an alarm message is sent to the operator of the live broadcast platform; if the image violation timestamp and the text violation timestamp are the same, and the violation probability is greater than the set violation threshold or there are violation keywords in the violated text paragraph, it is determined to be in violation, and an alarm message is sent to the operator of the live broadcast platform. Two violation determination situations are clarified. By comprehensively considering the image violation timestamp, the text violation timestamp, the violation probability, and the keywords in the violated text paragraph, violations can be identified comprehensively and accurately. This multi-dimensional determination method reduces the possibility of misjudgment and missed judgment, ensuring that as long as there are violation signs in the host's image or voice, they can be discovered in time. Once a violation is determined, an alarm message is immediately sent to the operator of the live broadcast platform, facilitating the operator to take timely measures, such as interrupting the live broadcast, warning or punishing the host, etc., effectively curbing the continuous spread of violations, maintaining the good order and healthy environment of the live broadcast platform, and guaranteeing the viewing experience of the audience.

[0072] Modify the data acquisition parameters based on the host's historical violation data.

[0073] Construct a platform-wide risk baseline matrix based on the historical violation data of each host, calculate the matrix elements, where the row dimension of the risk baseline matrix represents the violation risk base number of the violation type, and the column dimension represents the time window; obtain the real-time violation feature vector F of the live broadcast platform within the monitoring time window eral, Feral = [f1,..., fi ′ ,...], where where is the violation probability of the k'-th violation event corresponding to the i'-th violation type, γ time is the first time decay factor, ΔT is the monitoring time window, and i' is the number of the violation type.

[0074] Calculate the dynamic violation risk coefficient by combining the platform risk baseline matrix and the host influence coefficient of each host; determine the platform violation risk level based on the risk level mapping relationship, and match the corrected data collection parameters stored in the database based on the platform violation risk level. Construct a platform-wide risk baseline matrix, record violation situations from two dimensions of violation type and time window, and provide a systematic data framework for analyzing violation trends. By calculating the matrix elements, the risk levels of different violation types in each time window can be visually presented. Calculate the dynamic violation risk coefficient by combining the real-time violation feature vector and the host influence coefficient, and accurately evaluate the current violation risk level of the platform. Determine the platform violation risk level according to the risk level mapping relationship, and match the corrected data collection parameters accordingly, realizing the adaptive adjustment of the monitoring system. For example, when the violation risk is high, increase the data collection frequency to capture violation behaviors more timely; when the risk is low, reasonably reduce the collection frequency to save system resources, thereby improving the monitoring efficiency, optimizing resource allocation, and enhancing the effectiveness and sustainability of the monitoring system.

[0075] The platform-wide risk baseline matrix is denoted as

[0076] The calculation method of the matrix elements is as follows:

[0077]

[0078] where w i′j′ is the matrix element in the platform-wide risk baseline matrix, δ(C i′，j′ ) is the number of violations of the i'-th violation type in the time window j', is the maximum number of violations of the i'-th violation type, j' is the number of the time window, ξ is the second time decay factor, t now is the current time, t wg is the time of the most recent violation, and J is the total number of violation types; Sliding window update mechanism: Delete historical data older than 3 months at 0:00 every day. For the calculation of matrix elements, factors such as the number of violations of the violation type in the time window, the maximum number of violations, and time decay are comprehensively considered. Through this calculation method, the risk status of different violation types at different times can be accurately reflected, reflecting the historical trend and timeliness of violation behaviors, and providing an accurate data basis for evaluating the overall risk of the platform.

[0079] The method for obtaining the dynamic violation risk coefficient is as follows:

[0080]

[0081] where λ risk is the dynamic violation risk coefficient, is the transposed matrix of the full-platform risk baseline matrix M base and is the mean value of the influencing factors. The calculation of the dynamic violation risk coefficient combines the real-time violation feature vector, the transposed full-platform risk baseline matrix, and the mean value of the influencing factors, integrating the real-time violation situation with historical risk data and the influencing factors of the host, and can dynamically and comprehensively evaluate the current violation risk degree of the platform.

[0082] A monitoring system for Internet live broadcast violation behaviors of image and voice recognition, which is used for the monitoring method of Internet live broadcast violation behaviors of image and voice recognition as described above, such as Figure 2 shown, includes: a live stream data acquisition module, which is used to acquire the live stream data of each host from the live broadcast platform based on the determined data acquisition parameters and verify the data integrity. The live stream data includes the original video stream data and the original audio stream data; a data preprocessing module, which is used to process the live stream data through the determined data processing mode to obtain the preprocessed live stream data, including a video frame set and speech-to-text data; an image processing module, which is used to perform feature recognition on the video frame set by using the determined image violation processing model and output image violation marking information, including the violation type, violation probability, and image violation timestamp of each image frame; a text processing module, which is used to perform text processing on the speech-to-text data by using the determined text processing model and output text violation marking information, including the violation text paragraph and text violation timestamp; a violation judgment module, which is used to determine whether the host violates the regulations based on the image violation marking information and the text violation marking information and give an alarm when a violation occurs; a collection parameter correction module, which is used to correct the data collection parameters based on the host's historical violation data.

[0083] An electronic device, including: a processor; and a memory, in which computer program instructions are stored. When the computer program instructions are run by the processor, the processor executes the monitoring method of Internet live broadcast violation behaviors of image and voice recognition as described above.

[0084] A computer-readable storage medium is used to store a program, and when the program is executed by a processor, it implements the monitoring method of Internet live broadcast violation behaviors of image and voice recognition as described above.

[0085] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0086] The present invention is described with reference to the flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0087] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0089] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0090] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A monitoring method for Internet live broadcast violation behaviors of image and speech recognition, characterized in that, It includes the following steps: Obtain the live stream data of each anchor from the live streaming platform based on the determined data acquisition parameters and verify the data integrity. The live stream data includes the original video stream data and the original audio stream data; Process the live stream data through the determined data processing mode to obtain the preprocessed live stream data, including the video frame set and the speech-to-text data; Use the determined image violation processing model to perform feature recognition on the video frame set and output the image violation marking information, including the violation type, violation probability, and image violation timestamp of each image frame; Use the determined text processing model to perform text processing on the speech-to-text data and output the text violation marking information, including the violated text paragraph and the text violation timestamp; Determine whether the anchor violates the regulations based on the image violation marking information and the text violation marking information and give an alarm when a violation occurs; Modify the data acquisition parameters based on the historical violation data of the anchor.

2. The monitoring method for Internet live broadcast violation behaviors of image and voice recognition according to claim 1, characterized in that, Verify the data integrity of the live stream data of each anchor, including the following steps: Obtain the data integrity coefficient I of each host comp and the data format correctness index I form : Among them, D act is the actual amount of data returned, D exp is the amount of data that the request expects to obtain, N corr is the number of fields with correct formats in the data, N total is the total number of fields in the data; If the data integrity coefficient is greater than the determined data integrity threshold of the corresponding anchor and the data format correctness index is greater than the determined data format correctness threshold of the corresponding anchor, the data integrity of this anchor is normal; otherwise, the data integrity is abnormal, and the live stream data of this anchor is obtained again; Among them, the data acquisition parameters include the API application frequency and the data acquisition request interval time of the network crawler program; The data acquisition parameters, the data integrity threshold, the data format correctness threshold, the data processing mode, the image violation processing model, and the text processing model are all determined by obtaining the influencing factors corresponding to each anchor.

3. The monitoring method for Internet live broadcast violation behaviors of image and voice recognition according to claim 2, wherein, The determination methods of the data acquisition parameters, the data integrity threshold, and the data format correctness threshold are as follows: Obtain the set of preset influencing factor - data processing parameter mappings stored in the database, compare the influencing factor with each preset influencing factor one by one, calculate the similarity based on the Euclidean distance, determine the preset influencing factor corresponding to the minimum similarity, and obtain the corresponding data processing parameters from the database, including the data acquisition parameters, the data integrity threshold, and the data format correctness threshold; The data processing mode includes the video stream processing mode and the audio stream processing mode. Among them, the video stream processing mode includes the adaptive histogram equalization processing mode and the Gaussian filtering denoising processing mode, and the audio stream processing mode includes the first-level text processing mode and the second-level text processing mode; the image violation processing model includes the Mask R-CNN model and the Faster R-CNN model, and the text processing model includes the Transformer architecture text processing model and the keyword matching processing model; the determination methods of the data processing mode, the image violation processing model, and the text processing model are as follows: Compare the influencing factors corresponding to each anchor with the influence threshold; If the influencing factor is greater than the influence threshold, the video stream processing mode is the adaptive histogram equalization processing mode, the audio stream processing mode is the first-level text processing mode, the image violation processing model is the Mask R-CNN model, and the text processing model is the Transformer architecture text processing model; If the impact factor is not greater than the impact threshold, the video stream processing mode is the Gaussian filtering denoising processing mode, the audio stream processing mode is the second-level text processing mode, the image violation processing model is the Faster R-CNN model, and the text processing model is the keyword matching processing model.

4. The monitoring method for Internet live broadcast violation behaviors of image and voice recognition according to claim 2, characterized in that, The method for obtaining the impact factor is as follows: Obtain the influence data of the live streaming platform and the host influence data corresponding to each host, and determine the live streaming platform influence coefficient λ1 and the host influence coefficients of each host i is the number of the host; Based on the live streaming platform influence coefficient λ1 and the host influence coefficients of each host Obtain the initial influence factor Determine a correction factor based on the real-time live broadcast data of each anchor Perform real-time correction on the initial influence factor to obtain the influence factor e is the natural constant.

5. The monitoring method for Internet live broadcast violation behaviors of image and voice recognition according to claim 4, characterized in that, The influence coefficient λ1 of the live streaming platform, the method for obtaining it is as follows: λ1 = λ Sc + λ PF + λ B + λ TH + λ QL ; Among them, λ Sc is the market share coefficient, obtained based on the logistic regression model, and λ PF is the user word-of-mouth coefficient, obtained based on the calculation model of the average score and the standard deviation of the scores, and λ B is the media exposure coefficient, determined based on the analytic hierarchy process, and λ TH is the coefficient of the ratio of technological input to output, determined based on the technological innovation funds and the effect of achievement transformation, and λ QL is the user growth potential coefficient, determined based on the historical user growth rate; Host influence coefficient The acquisition method is as follows: Among them, is the host fan scale factor, is the host live broadcast activity factor, is the host content quality factor, and X is the total number of hosts in the live broadcast platform.

6. The monitoring method for Internet live broadcast violation behaviors of image and voice recognition according to claim 4, characterized in that, The correction factor, the method for obtaining it is as follows: Obtain the real-time data during the host's live broadcast, including the growth rate xZ of the number of viewers per unit time during the live broadcast, the keyword density ρ of the audience per unit time during the live broadcast gc and the total number of online people P in the current unit time during the live broadcast zx , and obtain the correction factor based on the correction factor calculation model Among them, is the average value of the number of online users during the host's historical live broadcast.

7. The monitoring method for Internet live broadcast violation behaviors of image and voice recognition according to claim 1, wherein Based on the image violation marking information and the text violation marking information, determine whether the anchor is in violation and issue an alarm when in violation, including the following steps: If the image violation timestamp is the same as the text violation timestamp, and the violation probability is greater than the set violation threshold and the violated text paragraph contains violated keywords, it is determined as a violation, and an alarm message is sent to the operator of the live streaming platform; If the image violation timestamp is the same as the text violation timestamp, and the violation probability is greater than the set violation threshold or the violated text paragraph contains violated keywords, it is determined as a violation, and an alarm message is sent to the operator of the live streaming platform.

8. The monitoring method for Internet live broadcast violation behaviors of image and voice recognition according to claim 4, characterized in that, Based on the historical violation data of the anchor, correct the data acquisition parameters, including the following steps: Construct a full-platform risk baseline matrix based on the historical violation data of each anchor, and calculate the matrix elements, where the row dimension of the risk baseline matrix represents the violation risk base of the violation type, and the column dimension represents the time window; Obtain the real-time violation feature vector of the live streaming platform within the monitored time window, and calculate the dynamic violation risk coefficient by combining the platform risk baseline matrix and the influence coefficient of each anchor; Determine the platform violation risk level based on the risk level mapping relationship, and match the corrected data acquisition parameters stored in the database based on the platform violation risk level.

9. The monitoring method for Internet live broadcast violations of image and voice recognition according to claim 8, characterized in that, The calculation method of the matrix elements is as follows: Among them, w i′j′ is the matrix element in the full-platform risk baseline matrix, and δ(C i′，j′ ) is the number of violations of the i'-th type of violation at time window j'. is the maximum number of violations of the i'-th type of violation, i' is the number of the type of violation, j' is the number of the time window, ξ is the time decay factor, t now is the current time, t wg is the time of the most recent violation, and J is the total number of types of violations. The method for obtaining the dynamic violation risk coefficient is: Among them, λ risk is the dynamic violation risk coefficient, F eral is the real-time violation feature vector, is the transposed matrix of the full-platform risk baseline matrix M base , and is the mean value of the influencing factors.

10. A monitoring system for internet live broadcast violation behaviors of image and voice recognition, which is used for the monitoring method of internet live broadcast violation behaviors of image and voice recognition according to any one of claims 1-9, characterized in that, Including: The live stream data acquisition module is used to obtain the live stream data of each anchor from the live streaming platform based on the determined data acquisition parameters and verify the data integrity. The live stream data includes the original video stream data and the original audio stream data; The data preprocessing module is used to process the live stream data through the determined data processing mode to obtain the preprocessed live stream data, including the video frame set and the speech-to-text data; The image processing module is used to perform feature recognition on the video frame set by using the determined image violation processing model, and output the image violation marking information, including the violation type, violation probability and image violation timestamp of each image frame; The text processing module is used to perform text processing on the speech-to-text data by using the determined text processing model, and output the text violation marking information, including the violated text paragraph and the text violation timestamp; The violation judgment module is used to determine whether the anchor is in violation based on the image violation marking information and the text violation marking information and issue an alarm when in violation; The acquisition parameter correction module is used to correct the data acquisition parameters based on the historical violation data of the anchor.

Images