A live multi-type violation early warning method and system based on incremental learning

Abstract

This invention relates to the field of live streaming technology, specifically disclosing an incremental learning-based method and system for early warning of multiple types of violations in live streaming. This invention acquires multimodal data from the live stream in real time and constructs a quantitative evaluation system integrating visual, acoustic, and semantic data. This system first extracts spatiotemporal violation components, abnormal audio components, and semantic violation components, and then dynamically weights and fuses them based on the live streaming scene to obtain a comprehensive confidence score. When the score is lower than a preset threshold, the system marks the sample as a low-confidence sample and initiates a few-shot learning mechanism to quickly generate an adaptive vector classification head. The model is then incrementally updated in a lightweight manner. Finally, by fusing the decision results of the original classification head and the incremental classification head, accurate and real-time early warnings for multiple types of violations in the live stream are achieved. This invention significantly improves the accuracy of violation detection and the model's rapid adaptability to new violation patterns through multimodal fusion and incremental learning.

Description

Technical Field

[0001] This invention relates to the field of live streaming technology, and in particular to an incremental learning method and system for early warning of multiple types of violations in live streaming. Background Technology

[0002] With the rapid development of the online live streaming industry, the amount of live streaming content is growing exponentially, and the potential for illegal content poses a serious challenge to platform content security. Traditional live streaming violation detection methods typically employ single-modal analysis, such as detecting visual violations solely through image recognition or audio violations solely through speech recognition. This approach is prone to misjudgments or omissions due to incomplete modal information.

[0003] However, violations in live streaming scenarios are dynamic and evolving, with new types of violations (such as emerging variants of inappropriate language and deceptive actions) constantly emerging. Traditional models struggle to adapt quickly to new violation patterns after training. Full retraining of the model is computationally expensive and has a long update cycle, failing to meet the real-time requirements of live streaming. Therefore, an incremental learning method for early warning of multiple types of violations in live streaming is needed to address these issues. Summary of the Invention

[0004] The purpose of this invention is to provide an incremental learning-based method for early warning of multiple types of violations in live streaming, including: Acquire multimodal data of a live stream, wherein the multimodal data includes visual feature information, acoustic feature information, and semantic feature information; Optical flow motion parameters and color histogram parameters are obtained based on the visual feature information, and spatiotemporal violation components are obtained based on the optical flow motion parameters and the color histogram parameters; The Mel frequency cepstral coefficients and fundamental frequency parameters are obtained based on the acoustic feature information, and the abnormal audio components are obtained based on the Mel frequency cepstral coefficients and fundamental frequency parameters. The text embedding vector is obtained as a semantic violation component based on the semantic feature information, and the semantic violation component, the spatiotemporal violation component and the abnormal audio component are weighted and fused to obtain a comprehensive confidence score. Determine whether the overall confidence score is lower than a preset threshold; If the overall confidence score is lower than a preset threshold, the multimodal data is marked as a low-confidence sample. A corresponding multimodal feature extractor is obtained based on the low-confidence sample. The low-confidence sample is then subjected to small-sample learning to obtain a small-sample adaptation vector classification head. The multimodal feature extractor is then incrementally updated based on the adaptation vector classification head to obtain an incremental update result. Based on the incremental update result, multiple types of violation warnings are issued for the live stream.

[0005] Preferably, the step of obtaining optical flow motion parameters and color histogram parameters based on the visual feature information, and obtaining the spatiotemporal violation component based on the optical flow motion parameters and color histogram parameters, includes: A continuous video frame sequence is obtained based on the visual feature information, and an optical flow field is obtained based on the continuous video frame sequence using the optical flow method. The instantaneous motion velocity and motion direction of the pixels are obtained based on the optical flow field. The instantaneous motion velocity and motion direction are statistically analyzed to obtain the optical flow motion parameters. A single-frame image is obtained based on the visual feature information, and the distribution frequency of pixels in each channel of the RGB color space in the single-frame image is extracted. A color histogram is constructed based on the distribution frequency, and the peak value, mean, and variance of the color histogram are calculated to obtain the color histogram parameters. A preset first spatiotemporal weight matrix is ​​obtained. The optical flow motion parameters are multiplied by the first spatiotemporal weight matrix to obtain motion violation components. The color histogram parameters are multiplied by the first spatiotemporal weight matrix to obtain color violation components. The first spatiotemporal weight matrix is ​​a preset weight vector that matches the dimensions of motion features and color features. Obtain a preset second spatiotemporal weight matrix, and perform a weighted sum based on the motion violation component, the color violation component, and the second spatiotemporal weight matrix. The second spatiotemporal weight matrix includes motion weights corresponding to the motion violation component and color weights corresponding to the color violation component, thereby obtaining the spatiotemporal violation component.

[0006] Preferably, the steps of obtaining Mel frequency cepstral coefficients and fundamental frequency parameters based on the acoustic feature information, and obtaining anomalous audio components based on the Mel frequency cepstral coefficients and fundamental frequency parameters, include: The audio signal within a preset window is obtained based on the acoustic feature information. The audio signal is pre-emphasized, framed, and windowed to obtain a pre-processed audio frame. The pre-processed audio frame is subjected to a fast Fourier transform to obtain the spectrum. The spectrum is passed through a Mel filter bank, and the logarithm of the filtered energy is taken before a discrete cosine transform is performed to obtain the Mel frequency cepstral coefficients. Based on the audio signal, candidate fundamental frequency values ​​are obtained, and a continuous fundamental frequency trajectory is extracted from the candidate fundamental frequency values ​​using a dynamic programming algorithm. The fundamental frequency trajectory is then smoothed to obtain the fundamental frequency parameters. A preset audio violation detection model is obtained, wherein the audio violation detection model is a binary classification neural network model trained based on a large number of labeled violation and non-violation audio samples. The Mel frequency cepstral coefficients and the fundamental frequency parameters are input into the audio violation detection model to obtain the abnormal audio probability value output by the audio violation detection model. The abnormal audio probability value is used as the abnormal audio component.

[0007] Preferably, the step of obtaining the text embedding vector as a semantic violation component based on the semantic feature information, and then performing a weighted fusion of the semantic violation component, the spatiotemporal violation component, and the anomalous audio component to obtain a comprehensive confidence score includes: Based on the semantic feature information, obtain the real-time subtitle text or speech recognition text in the live stream, and perform word segmentation and stop word removal on the real-time subtitle text or speech recognition text to obtain preprocessed text; A pre-trained language representation model is obtained, the pre-processed text is input into the pre-trained language representation model, and the text embedding vector output by the pre-trained language representation model is obtained. The text embedding vector is used as a semantic violation component. Obtain a dynamic weight allocation strategy, wherein the dynamic weight allocation strategy is a mapping table that maps live streaming scene types to weight sets, and each weight set contains a semantic weight, a spatiotemporal weight and an audio weight. Identify the scene type of the current live stream, and obtain the semantic weight corresponding to the semantic violation component, the spatiotemporal weight corresponding to the spatiotemporal violation component, and the audio weight corresponding to the abnormal audio component from the dynamic weight allocation strategy according to the scene type; Based on the semantic weight, the spatiotemporal weight, and the audio weight, the semantic violation component, the spatiotemporal violation component, and the anomalous audio component are weighted and summed to obtain a comprehensive confidence score.

[0008] Preferably, the step of performing small-sample learning on the low-confidence samples to obtain a small-sample adaptive vector classification head, and incrementally updating the multimodal feature extractor based on the adaptive vector classification head to obtain the incremental update result includes: Obtain multimodal data of samples labeled as low-confidence samples, and use the multimodal data of the low-confidence samples as a support set; The support set is input into the multimodal feature extractor to obtain the feature vector of each sample in the support set; Obtain the violation type label for each sample in the support set, and group the feature vectors according to the violation type label, and calculate the prototype vector for each group of feature vectors; An adaptation vector classification head is constructed based on the prototype vector, wherein the adaptation vector classification head is used to calculate the distance between the feature vector of the sample to be classified and the prototype vector; A preset incremental update strategy is obtained. The adaptation vector classification head is integrated into the output of the multimodal feature extractor according to the incremental update strategy. Only the parameters of the last two layers of the multimodal feature extractor are fine-tuned, while the parameters of the remaining layers are fixed, to obtain the incremental update result.

[0009] Preferably, the step of providing multi-type violation warnings for the live stream based on the incremental update results includes: Obtain the incremental update result, wherein the incremental update result includes the updated multimodal feature extractor and the newly added adaptation vector classification head; The updated multimodal feature extractor is used to obtain the feature vector of the new multimodal data in the live stream. Based on the multimodal data, obtain the corresponding preset original classification head, and input the feature vector into the preset original classification head and the newly added adaptation vector classification head respectively to obtain the original classification result and the adaptation classification result; A decision fusion rule is obtained, wherein the decision fusion rule is as follows: if the original classification result and the adapted classification result are consistent, then the original classification result is taken as the final classification result; if they are inconsistent, then the result with higher confidence is selected as the final classification result, and the original classification result and the adapted classification result are fused according to the decision fusion rule to obtain the final classification result; Based on the final classification result, it is determined whether there is any violation in the live stream. If the final classification result exceeds the preset violation threshold, an early warning message is generated, and multiple types of violation warnings are issued to the live stream based on the early warning message.

[0010] This application also provides an incremental learning-based live streaming multi-type violation early warning system, including: The first acquisition module is used to acquire multimodal data of the live stream, wherein the multimodal data includes visual feature information, acoustic feature information and semantic feature information; The second acquisition module is used to acquire optical flow motion parameters and color histogram parameters based on the visual feature information, and to acquire spatiotemporal violation components based on the optical flow motion parameters and the color histogram parameters. The third acquisition module is used to acquire Mel frequency cepstral coefficients and fundamental frequency parameters based on the acoustic feature information, and to acquire abnormal audio components based on the Mel frequency cepstral coefficients and fundamental frequency parameters. The fourth acquisition module is used to acquire the text embedding vector as a semantic violation component based on the semantic feature information, and to perform weighted fusion based on the semantic violation component, the spatiotemporal violation component and the abnormal audio component to obtain a comprehensive confidence score; The judgment module is used to determine whether the overall confidence score is lower than a preset threshold; If the overall confidence score is lower than a preset threshold, the multimodal data is marked as a low-confidence sample. A corresponding multimodal feature extractor is obtained based on the low-confidence sample. The low-confidence sample is then subjected to small-sample learning to obtain a small-sample adaptation vector classification head. The multimodal feature extractor is then incrementally updated based on the adaptation vector classification head to obtain an incremental update result. Based on the incremental update result, multiple types of violation warnings are issued for the live stream.

[0011] Preferably, the second acquisition module includes: The first acquisition unit is used to acquire a continuous video frame sequence based on the visual feature information, acquire an optical flow field based on the continuous video frame sequence using the optical flow method, acquire the instantaneous motion speed and motion direction of the pixel based on the optical flow field, and statistically analyze the instantaneous motion speed and motion direction to obtain optical flow motion parameters. The second acquisition unit is used to acquire a single frame image based on the visual feature information, extract the distribution frequency of pixels in each channel of the RGB color space in the single frame image, construct a color histogram based on the distribution frequency, and calculate the peak value, mean and variance of the color histogram to obtain the color histogram parameters. The third acquisition unit is used to acquire a preset first spatiotemporal weight matrix, perform a dot product operation between the optical flow motion parameters and the first spatiotemporal weight matrix to obtain motion violation components, and perform a dot product operation between the color histogram parameters and the first spatiotemporal weight matrix to obtain color violation components. The first spatiotemporal weight matrix is ​​a preset weight vector that matches the dimensions of motion features and color features. The fourth acquisition unit is used to acquire a preset second spatiotemporal weight matrix and perform a weighted summation based on the motion violation component, the color violation component, and the second spatiotemporal weight matrix, wherein the second spatiotemporal weight matrix includes motion weights corresponding to the motion violation component and color weights corresponding to the color violation component, to obtain the spatiotemporal violation component.

[0012] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0013] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0014] The beneficial effects of this application are as follows: This invention acquires multimodal data from live streams in real time, constructs a multi-dimensional violation detection system, and dynamically updates the model. First, it collects visual, acoustic, and semantic feature information from the live stream. It obtains spatiotemporal violation components by fusing optical flow motion parameters and color histogram parameters, and abnormal audio components by fusing Mel-frequency cepstral coefficients and fundamental frequency parameters. Semantic violation components are obtained through a pre-trained language model. Then, weights are dynamically assigned according to the live stream scene type, and the three-modal violation components are weighted and fused to obtain a comprehensive confidence score. When the confidence score is lower than a preset threshold, the sample is marked as a low-confidence sample and used as a support set. An adaptive vector classification head is constructed through few-shot learning, and only the last two layers of the multimodal feature extractor are fine-tuned to achieve rapid incremental updates of the model. Finally, the decision results of the original classification head and the newly added classification head are fused to generate the final warning information. This invention enables the model to quickly adapt to newly emerging violation types through an incremental learning mechanism, without requiring full retraining, balancing detection accuracy and real-time updates, and effectively improving the intelligent level of live stream content security monitoring. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a method flow according to an embodiment of this application.

[0016] Figure 2 This is a schematic diagram of the system structure according to an embodiment of this application.

[0017] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] like Figure 1 As shown, this application provides an incremental learning-based method for early warning of multiple types of violations in live streaming, including: S1. Acquire multimodal data of the live stream, wherein the multimodal data includes visual feature information, acoustic feature information and semantic feature information. The visual feature information is obtained by acquiring the live video stream in real time through a video capture card. The acoustic feature information is obtained by acquiring audio signals in real time through a microphone array. The semantic feature information is obtained through the live streaming platform's subtitle interface or speech recognition module. S2. Obtain optical flow motion parameters and color histogram parameters based on the visual feature information, and obtain spatiotemporal violation components based on the optical flow motion parameters and the color histogram parameters; S3. Obtain the Mel frequency cepstral coefficients and fundamental frequency parameters based on the acoustic feature information, and obtain the abnormal audio components based on the Mel frequency cepstral coefficients and fundamental frequency parameters; S4. Obtain the text embedding vector as the semantic violation component based on the semantic feature information, and perform weighted fusion based on the semantic violation component, the spatiotemporal violation component and the abnormal audio component to obtain the comprehensive confidence score; S5. Determine whether the overall confidence score is lower than a preset threshold; S6. If the overall confidence score is lower than a preset threshold, the multimodal data is marked as a low-confidence sample. A corresponding multimodal feature extractor is obtained based on the low-confidence sample. The low-confidence sample is subjected to small-sample learning to obtain a small-sample adaptation vector classification head. The multimodal feature extractor is incrementally updated based on the adaptation vector classification head to obtain an incremental update result. Based on the incremental update result, a multi-type violation warning is issued for the live stream.

[0020] As described in steps S1-S6 above, due to the real-time, diverse, and complex nature of live streaming content, single-modal detection methods struggle to accurately identify complex violations, and the models lack adaptability to novel violation patterns. This invention acquires multimodal data from the live stream, including visual, acoustic, and semantic features. This step is fundamental to the entire violation warning method; all violation feature analyses are based on real, real-time live stream data. Physically speaking, only by comprehensively acquiring the original feature data of each dimension of the live stream can data support be provided for the subsequent extraction of each modal violation component, avoiding detection blind spots caused by missing data dimensions. In terms of technical implementation, visual feature information is obtained through real-time acquisition and feature extraction of video frames from the live stream, covering visual attributes such as pixel size, motion, and color. Acoustic feature information is obtained through real-time acquisition and analog-to-digital conversion of audio signals from the live stream, including acoustic attributes such as frequency, amplitude, and fundamental frequency. Semantic feature information is obtained through real-time acquisition of subtitles or speech-to-text conversion from the live stream, covering the language and text content of the live stream. For example, in entertainment live streams, the streamer's video can be acquired in real-time as visual feature information, the streamer's voice and background music as acoustic feature information, and the streamer's dialogue and interactive text as semantic feature information. The technical effect of this step is that it builds a multi-dimensional data foundation for subsequent violation feature analysis, breaking the limitations of single-modal data, and allowing subsequent violation detection to be carried out simultaneously from three dimensions: visual, acoustic, and semantic, providing data assurance for detection accuracy.

[0021] The optical flow motion parameters and color histogram parameters are obtained based on visual feature information, and the spatiotemporal violation components are obtained based on the optical flow motion parameters and color histogram parameters. This step is a quantitative extraction of the visual dimension violation features. Its physical significance is that visual violations in live streams are often reflected in the motion features and color features of the image. For example, illegal body movements will cause abnormal movement of pixels in the image, and illegal image content will show abnormal color distribution. By calculating and fusing the optical flow motion parameters and color histogram parameters, the visual dimension violation features can be transformed into quantifiable spatiotemporal violation components, so as to achieve accurate identification of visual violations. In terms of technical implementation, firstly, based on the continuous video frame sequence in the visual feature information, the optical flow field is obtained through the optical flow method. Then, the instantaneous motion speed and direction of the pixels are statistically analyzed from the optical flow field to obtain the optical flow motion parameters. These parameters can accurately characterize the motion state of the target in the video image. For example, when there is rapid and irregular limb movement in the live broadcast, the optical flow motion parameters will show obvious anomalies. Secondly, based on the single-frame image in the visual feature information, the distribution frequency of each channel pixel in the RGB color space is extracted, a color histogram is constructed, and its peak value, mean, and variance are calculated to obtain the color histogram parameters. These parameters can characterize the color distribution characteristics of the image. For example, when there is an irregular high saturation and high contrast color distribution in the live broadcast, the color histogram parameters will show anomalies. Subsequently, a preset first spatiotemporal weight matrix is ​​obtained. This matrix is ​​a weight vector matched with the motion and color feature dimensions. The optical flow motion parameters and color histogram parameters are multiplied by this matrix to obtain the motion violation component and the color violation component. Then, the motion violation component and the color violation component are weighted and summed using a second spatiotemporal weight matrix that includes motion weights and color weights, finally obtaining the spatiotemporal violation component. The technical effect of this step is that it transforms the abstract motion and color features in visual feature information into quantifiable and computable spatiotemporal violation components, realizing the accurate quantification of visual dimension violations. At the same time, through the weighted operation of the two-layer weight matrix, the weights of motion and color features can be flexibly adjusted according to the actual needs of visual violation detection. For example, in a dance live broadcast scenario, the motion weight can be appropriately increased to more accurately identify visual violations related to body movements. In a static image display live broadcast, the color weight can be appropriately increased to more accurately identify visual violations related to image color, improving the flexibility and adaptability of visual dimension violation detection.

[0022] The Mel frequency cepstral coefficients and fundamental frequency parameters are obtained based on acoustic feature information, and abnormal audio components are obtained based on the Mel frequency cepstral coefficients and fundamental frequency parameters. This step is a quantitative extraction of acoustic dimension violation features. Its physical significance is that acoustic violations in live streams will be reflected in the spectral and fundamental frequency features of the audio. For example, violation speech content will be accompanied by abnormal tone and speed, and violation audio effects will have abnormal spectral distribution. By calculating the Mel frequency cepstral coefficients and fundamental frequency parameters, and combining them with the judgment of the audio violation discrimination model, the acoustic dimension violation features can be transformed into quantifiable abnormal audio components, so as to achieve accurate identification of acoustic violations. In terms of technical implementation, the audio signal within a preset window is first extracted from the acoustic feature information. This signal undergoes pre-emphasis, framing, and windowing preprocessing to obtain a preprocessed audio frame. A Fast Fourier Transform (FFT) is then performed on the preprocessed audio frame to obtain its spectrum. After passing the spectrum through a Mel filter bank, the logarithm of the filtered energy is taken, followed by a Discrete Cosine Transform (DCT) to obtain the Mel frequency cepstral coefficients. These coefficients are crucial for representing the spectral characteristics of the audio signal, accurately reflecting attributes such as timbre and spectral distribution. Next, candidate fundamental frequencies are obtained from the audio signal. A dynamic programming algorithm is used to extract continuous fundamental frequency trajectories, which are then smoothed to obtain the fundamental frequency parameter. This parameter characterizes the pitch variation of the audio; for example, when a broadcaster uses abusive language or shouting, the fundamental frequency parameter will exhibit obvious abnormal fluctuations. Subsequently, the Mel frequency cepstral coefficients and the fundamental frequency parameter are input into a preset audio violation discrimination model. This model is a binary classification neural network model trained on a large number of labeled violation and non-violation audio samples. The model outputs the corresponding abnormal audio probability value, which is the abnormal audio component. The technical effect of this step is that, by using acoustic feature extraction algorithms and pre-trained binary classification neural network models, the abstract spectrum and fundamental frequency features in acoustic feature information are transformed into quantifiable abnormal audio components, achieving accurate quantification of acoustic dimension violations. At the same time, the audio violation discrimination model is trained based on a large number of labeled samples and has a strong acoustic violation recognition capability, which can effectively identify various acoustic violations such as verbal abuse, illegal sound effects, and abnormal intonation, thus improving the accuracy of acoustic dimension violation detection.

[0023] The text embedding vector is obtained based on semantic feature information as a semantic violation component. The semantic violation component, spatiotemporal violation component, and abnormal audio component are then weighted and fused to obtain a comprehensive confidence score. This step is the quantitative extraction of semantic dimension violation features and the fusion evaluation of the violation components of the visual, acoustic, and semantic dimensions. Its physical significance is that semantic violations in live streams are mainly reflected in the language text content, such as illegal wording, deceptive expressions, and abusive text. By converting the text content into text embedding vectors through a pre-trained language representation model, the semantic violation features can be quantified. By combining the dynamic weight allocation of live stream scene type, the violation components of the three dimensions are weighted and fused to obtain a comprehensive confidence score that reflects the degree of violation in the live stream, thus achieving an overall judgment of the violation status of the live stream. In terms of technical implementation, the system first obtains real-time subtitle text or speech recognition text from the live stream from semantic feature information. This text undergoes preprocessing, including word segmentation and stop word removal, to obtain preprocessed text. This preprocessed text is then input into a pre-trained language representation model, which outputs a corresponding text embedding vector. This vector represents the semantic violation component, accurately representing the semantic features of the text content. For example, when a violation word appears in the text, the text embedding vector will exhibit features significantly different from normal semantics. Next, a dynamic weight allocation strategy is obtained. This strategy maps live stream scene types to weight sets. Each weight set includes semantic weights, spatiotemporal weights, and audio weights. The current live stream scene type is first identified, such as e-commerce live stream, game live stream, or entertainment live stream. Then, corresponding weights are matched from the dynamic weight allocation strategy based on the scene type. For example, in an e-commerce live stream scene, the semantic weight can be appropriately increased to more accurately identify semantic violations such as false advertising and fraudulent language. In a game live stream scene, the audio weight can be appropriately increased to more accurately identify acoustic violations such as verbal abuse. Finally, based on the matched semantic weights, spatiotemporal weights, and audio weights, the semantic violation components, spatiotemporal violation components, and anomalous audio components are weighted and summed to obtain a comprehensive confidence score. The technical benefits of this step are twofold: firstly, it achieves accurate quantification of semantic-dimensional violations, compensating for the shortcomings of visual and acoustic dimensions in semantic violation detection; secondly, it enables intelligent fusion of multimodal violation components through a dynamic weight allocation strategy, allowing the comprehensive confidence score to accurately reflect the actual degree of violation in the live stream according to the characteristics of different live streaming scenarios. This avoids the problem of poor scenario adaptability caused by fixed-weight fusion, providing a scientific and accurate basis for subsequent screening of low-confidence samples.

[0024] Determining whether the overall confidence score is below a preset threshold is the trigger point for the entire incremental learning mechanism. Its physical significance lies in the fact that the overall confidence score is a quantitative representation of the overall violation severity of the live stream. When this score is below the preset threshold, it indicates that the current model's violation judgment result for the live stream data has low reliability. This could be due to the emergence of a new violation pattern that the model has not learned, or it could be due to ambiguity in the judgment result after multimodal feature fusion. In this case, the sample needs to be marked as a low-confidence sample, and the subsequent incremental learning mechanism should be initiated to optimize the model. Technically, the preset threshold is a value determined based on a large number of labeled live stream samples, combined with the violation detection requirements of different live streaming scenarios. This value can effectively distinguish between high-confidence and low-confidence judgment samples. For example, in a general live streaming scenario, the preset threshold can be set to 0.7. When the overall confidence score is below 0.7, it is judged as a low-confidence sample. The technical effect of this step is that, through clear numerical judgment criteria, it achieves accurate screening of low-confidence samples, sets clear and executable triggering conditions for the subsequent launch of incremental learning mechanisms, avoids meaningless model updates, and ensures the relevance and effectiveness of incremental learning.

[0025] If the overall confidence score is lower than a preset threshold, the multimodal data is marked as a low-confidence sample. The currently used multimodal feature extractor is obtained based on the low-confidence sample. The low-confidence sample is then used for few-shot learning to obtain a small-sample adaptation vector classifier head. The multimodal feature extractor is then incrementally updated based on the adaptation vector classifier head to obtain the incremental update result. Based on the incremental update result, multi-type violation warnings are issued for the live stream. This step is the core implementation step of the incremental learning mechanism of this invention. Its physical significance lies in the fact that, when faced with new violation patterns reflected by low-confidence samples, there is no need to retrain the multimodal feature extractor. Instead, an adaptation vector classifier head is constructed through few-shot learning, and only a lightweight fine-tuning of the multimodal feature extractor is performed, achieving rapid incremental updates of the model. This allows the updated model to accurately identify new violation patterns. Finally, through the decision fusion of the original classifier head and the adaptation classifier head, the final accurate warning for violations in the live stream is achieved. In terms of technical implementation, the labeled low-confidence samples are first used as the support set and input into the corresponding multimodal feature extractor to obtain the feature vector of each sample in the support set. Then, the feature vectors are grouped according to the violation type label of the samples, and the prototype vector of each group of feature vectors is calculated. An adaptation vector classification head is constructed based on the prototype vectors. This classification head can be used to calculate the distance between the feature vector of the sample to be classified and the prototype vector to achieve the classification judgment of new violation types. Subsequently, a preset incremental update strategy is obtained, and the adaptation vector classification head is integrated into the output of the multimodal feature extractor. Only the parameters of the last two layers of the multimodal feature extractor are fine-tuned, while the parameters of the remaining layers are fixed to obtain the incremental update result. This update method only adjusts the shallow parameters of the model, avoiding the high computational consumption of full retraining. Finally, based on the updated multimodal feature extractor and the newly added adaptive vector classifier from the incremental update results, the feature vectors of the new multimodal data in the live stream are obtained. These feature vectors are then input into the preset original classifier and the newly added adaptive vector classifier, respectively, to obtain the original classification result and the adapted classification result. The two classification results are then fused according to the decision fusion rule: if the results are consistent, they are directly used as the final classification result; if the results are inconsistent, the result with higher confidence is selected as the final classification result. If the final classification result exceeds the preset violation threshold, a warning message is generated, and multiple types of violation warnings are issued. The technical advantage of this step is that, through few-sample learning and lightweight parameter fine-tuning, rapid incremental updates of the multimodal feature extractor are achieved. Compared to full retraining, this significantly reduces the computational power requirement and time cost of model updates, enabling rapid adaptation to new violation patterns appearing in live streaming scenarios. Furthermore, the decision fusion of the original and adapted classifiers balances the model's ability to identify both traditional and new violation patterns, resulting in more accurate violation warnings.

[0026] In one embodiment, step S2, which involves obtaining optical flow motion parameters and color histogram parameters based on the visual feature information, and obtaining spatiotemporal violation components based on the optical flow motion parameters and color histogram parameters, includes: S21. Obtain a continuous video frame sequence based on the visual feature information, and obtain an optical flow field based on the continuous video frame sequence using the optical flow method. Obtain the instantaneous motion speed and motion direction of the pixel based on the optical flow field, and statistically analyze the instantaneous motion speed and motion direction to obtain optical flow motion parameters. S22. Obtain a single-frame image based on the visual feature information, extract the distribution frequency of pixels in each channel of the RGB color space in the single-frame image, construct a color histogram based on the distribution frequency, and calculate the peak value, mean and variance of the color histogram to obtain the color histogram parameters. S23. Obtain a preset first spatiotemporal weight matrix, perform a dot product operation between the optical flow motion parameters and the first spatiotemporal weight matrix to obtain motion violation components, and perform a dot product operation between the color histogram parameters and the first spatiotemporal weight matrix to obtain color violation components, wherein the first spatiotemporal weight matrix is ​​a preset weight vector that matches the dimensions of motion features and color features. S24. Obtain a preset second spatiotemporal weight matrix, and perform a weighted summation based on the motion violation component, the color violation component, and the second spatiotemporal weight matrix, wherein the second spatiotemporal weight matrix includes the motion weight corresponding to the motion violation component and the color weight corresponding to the color violation component, to obtain the spatiotemporal violation component.

[0027] As described in steps S21-S24 above, visual violations in live streams are not manifested by a single feature, but are reflected simultaneously in both dynamic motion features and static color features of the image. For example, illegal body movements in a live stream can cause abnormal movement trajectories of pixels in the image, and illegal content will exhibit different color distribution patterns than normal live stream images. This requires that the extraction of visual violation features must cover both dynamic and static dimensions. Therefore, this invention obtains a continuous video frame sequence based on visual feature information, and obtains an optical flow field based on the continuous video frame sequence using optical flow method. Based on the optical flow field, it obtains the instantaneous motion speed and direction of the pixels, and statistically analyzes the instantaneous motion speed and direction to obtain optical flow motion parameters. This step is a quantitative extraction of the visual dynamic motion features of the live stream. Its physical significance is that dynamic violations in the live stream image are directly manifested as abnormal motion states of pixels. By capturing the instantaneous motion speed and direction of pixels using optical flow method and performing statistical analysis, the dynamic motion features of the image can be transformed into quantified optical flow motion parameters, achieving accurate capture of dynamic visual violation features. In terms of technical implementation, visual feature information is acquired by the real-time video frame acquisition module of the live stream. First, a continuous video frame sequence is extracted from the visual feature information. The frame interval and acquisition duration of this sequence are set according to the real-time requirements of the live stream. Generally, a short sequence of 10-30 consecutive frames is selected to balance computing power and feature effectiveness. Then, the classic dense optical flow method is used to calculate the optical flow field of the continuous video frame sequence. This algorithm can obtain the optical flow vector of each pixel in the video frame by solving the optical flow constraint equation. The magnitude of the optical flow vector is the instantaneous motion velocity of the pixel, and the direction of the optical flow vector is the motion direction of the pixel. Subsequently, the instantaneous motion velocity and motion direction of all pixels in the whole frame are statistically analyzed, and the mean, variance, extreme values ​​of the velocity and the distribution ratio of the direction are calculated. After integrating these statistical quantities, the optical flow motion parameters are obtained. For example, in entertainment live streaming scenarios, if a streamer makes large, irregular body movements that violate regulations, the instantaneous motion speed of the corresponding pixels in the image will show obvious extreme values, and the distribution of motion directions will also exhibit disorder. These anomalies will be directly reflected in the statistical results of optical flow motion parameters, which can be used to quickly capture the characteristics of such dynamic visual violations. The technical effect of this step is that it uses optical flow to achieve refined and quantitative extraction of dynamic motion features from live streaming images, overcoming the problem of insufficient representation of dynamic features in traditional visual feature extraction. The obtained optical flow motion parameters can accurately reflect the motion state of the target in the image, providing a precise quantitative basis for the subsequent calculation of dynamic motion violation components. At the same time, selecting short frame sequences for optical flow field calculation can effectively control computing power consumption and adapt to the application requirements of real-time detection in live streaming.

[0028] The process involves acquiring a single-frame image based on visual feature information and extracting the distribution frequency of pixels in each channel of the RGB color space. A color histogram is then constructed based on these distribution frequencies, and the peak value, mean, and variance of the color histogram are calculated to obtain color histogram parameters. This step quantifies and extracts the static color features of the live stream. Its physical significance lies in the fact that static violations in the live stream will manifest as abnormal color distribution. For example, violating content often exhibits excessive concentration of pixel values ​​in a certain RGB channel or abnormal dispersion in color distribution. By constructing a color histogram in the RGB color space and calculating its statistical features, the static color features of the image can be transformed into quantified color histogram parameters, enabling accurate capture of static visual violations. In terms of technical implementation, key single-frame images are selected from the visual feature information obtained by the video frame acquisition module. Generally, the middle frame of a continuous video frame sequence is selected to ensure the representativeness of the image. Then, the pixels of the single-frame image are mapped to the RGB color space, and the distribution frequency of each pixel value in the red, green, and blue channels is statistically analyzed. With the pixel value as the horizontal axis and the distribution frequency as the vertical axis, color histograms for the three channels are constructed. Subsequently, statistical calculations are performed on the color histogram of each channel to obtain the peak value (the most frequently occurring pixel value), the mean value (the weighted average of pixel values), and the variance (the dispersion of pixel value distribution). The peak value, mean value, and variance of the three channels are integrated to obtain the color histogram parameters. For example, when illegal pornographic or violent scenes appear in a live broadcast, the image often shows a high frequency of high pixel value distribution in the red channel. The peak value of its color histogram will be significantly higher, and the variance will show abnormal dispersion. These abnormal color distributions will be directly reflected in the color histogram parameters, which can effectively capture the characteristics of such static visual violations. The technical effect of this step is that it completes the quantitative extraction of static color features of the live broadcast in the RGB color space. The resulting color histogram parameters can comprehensively represent the color distribution pattern of the image from three dimensions: peak value, mean value, and variance. This makes up for the shortcomings of only extracting dynamic motion features for static visual violation detection. At the same time, the RGB color space is the basic color space for video processing. Its computational complexity is low, which can effectively improve the efficiency of feature extraction and adapt to the needs of real-time detection in live broadcasts.

[0029] A preset first spatiotemporal weight matrix is ​​obtained. The optical flow motion parameters are multiplied by the first spatiotemporal weight matrix to obtain the motion violation component. The color histogram parameters are multiplied by the first spatiotemporal weight matrix to obtain the color violation component. The first spatiotemporal weight matrix is ​​a pre-set weight vector that matches the dimensions of motion features and color features. This step is the first quantization and weighting of the extracted optical flow motion parameters and color histogram parameters. Its physical significance is that both optical flow motion parameters and color histogram parameters are composed of multiple statistics. Different statistics have different importance in representing visual violation features. By performing a dot product operation with the first spatiotemporal weight matrix that matches the dimensions, the statistics can be weighted differently, highlighting the statistical features that are more valuable for representing violations, and realizing the transformation from the original feature parameters to a single violation component. In terms of technical implementation, the first spatiotemporal weight matrix is ​​a weight vector pre-trained using gradient descent based on a large number of labeled visual violation and non-violation samples. Its dimensions perfectly match the statistical dimensions of the optical flow motion parameters and color histogram parameters. For example, if the optical flow motion parameters include three statistics: mean velocity, velocity variance, and directional distribution percentage, the first spatiotemporal weight matrix is ​​a 3-dimensional weight vector. The weight value of each dimension corresponds to the importance of a statistic, and the weight values ​​are set to a range of 0-1. The sum of the weight values ​​of all dimensions is 1 to ensure the consistency of the dimensions of the weighted parameters. Subsequently, each statistic of the optical flow motion parameters is used as a row vector and multiplied by the first spatiotemporal weight matrix. The resulting scalar value is the motion violation component. Similarly, each statistic of the color histogram parameters is multiplied by the first spatiotemporal weight matrix, and the resulting scalar value is the color violation component. For example, in optical flow motion parameters, velocity variance is more valuable for representing dynamic violations, and its corresponding weight value is set to a higher value. After dot product operation, the anomaly of velocity variance will be more obviously reflected in the motion violation component. The technical effect of this step is that, through the differentiated weighting of the first spatiotemporal weight matrix, feature selection and quantization enhancement of the original feature parameters are achieved, so that the obtained motion violation component and color violation component can more accurately represent the corresponding visual violation features. At the same time, the computational complexity of dot product operation is low, which can ensure the efficiency of feature processing, and the pre-trained weight matrix can be adapted to general live broadcast visual violation detection scenarios, laying the foundation for subsequent feature fusion.

[0030] A preset second spatiotemporal weight matrix is ​​obtained, and a weighted sum is performed based on the motion violation component, the color violation component, and the second spatiotemporal weight matrix. The second spatiotemporal weight matrix includes motion weights corresponding to the motion violation component and color weights corresponding to the color violation component to obtain the spatiotemporal violation component. This step is a second fusion and weighting of the motion violation component and the color violation component, finally obtaining the comprehensive spatiotemporal violation component of the visual dimension. Its physical significance is that in different live streaming scenarios, the importance of dynamic motion features and static color features for visual violation detection varies. By performing a weighted sum using the second spatiotemporal weight matrix containing motion weights and color weights, the fusion ratio of the two types of components can be adjusted according to the actual needs of the live streaming scenario, realizing personalized and scenario-based representation of visual violation features. In terms of technical implementation, the second spatiotemporal weight matrix is ​​a two-dimensional weight vector pre-defined according to the visual violation detection requirements of different live streaming scenarios. It includes two weight values: motion weight and color weight. The weight values ​​also range from 0 to 1, and the sum of the two weight values ​​is 1. For live streaming scenarios with dynamic images, such as dance and outdoor performances, the motion weight is set to a higher value, such as 0.7, and the color weight is set to 0.3. For live streaming scenarios with static images, such as calligraphy and painting performances and exhibitions, the color weight is set to a higher value, such as 0.8, and the motion weight is set to 0.2. Then, the spatiotemporal violation component is obtained by weighted summation according to the formula: spatiotemporal violation component = motion violation component × motion weight + color violation component × color weight. This component is a scalar between 0 and 1, and the higher the value, the higher the degree of visual violation in the live streaming image. The technical effect of this step is that it realizes the scene-based fusion of motion and color violation components through the second spatiotemporal weight matrix, so that the obtained spatiotemporal violation components can better fit the visual violation detection needs of different live broadcast scenarios, improve the scene adaptability of visual violation feature representation, and the weighted summation calculation method is simple and efficient, without increasing the computing power burden of real-time live broadcast detection. The resulting scalar form of spatiotemporal violation components also provides a unified quantitative standard for subsequent weighted fusion with acoustic and semantic violation components.

[0031] In one embodiment, step S3, which involves obtaining Mel-frequency cepstral coefficients and fundamental frequency parameters based on the acoustic feature information, and obtaining anomalous audio components based on the Mel-frequency cepstral coefficients and fundamental frequency parameters, includes: S31. Obtain the audio signal within a preset window based on the acoustic feature information, perform pre-emphasis, framing and windowing processing on the audio signal to obtain a pre-processed audio frame, perform fast Fourier transform on the pre-processed audio frame to obtain the spectrum, pass the spectrum through a Mel filter bank, and perform discrete cosine transform on the logarithm of the filtered energy to obtain Mel frequency cepstral coefficients. S32. Obtain candidate fundamental frequency values ​​based on the audio signal, extract continuous fundamental frequency trajectories from the candidate fundamental frequency values ​​using a dynamic programming algorithm, smooth the fundamental frequency trajectories, and obtain fundamental frequency parameters. S33. Obtain a preset audio violation detection model, wherein the audio violation detection model is a binary classification neural network model trained based on a large number of labeled violation and non-violation audio samples. The model adopts a shallow convolutional neural network structure, the input layer dimension is the sum of the Mel frequency cepstral coefficient dimension and the fundamental frequency parameter dimension, and the output layer uses the Sigmoid activation function to output abnormal audio probability values ​​between 0 and 1. The Mel frequency cepstral coefficient and the fundamental frequency parameter are input into the audio violation detection model to obtain the abnormal audio probability values ​​output by the audio violation detection model, and the abnormal audio probability values ​​are used as abnormal audio components.

[0032] As described in steps S31-S33 above, acoustic violations in live streams are manifested through physical characteristics such as the spectral distribution and pitch changes of audio signals. For example, abusive or shouting violations will exhibit abnormal fluctuations in the fundamental frequency, and violating sound effects and noise will present spectral characteristics different from normal live stream audio. Since the original acoustic feature information is a continuous analog audio signal, it cannot be directly used for violation determination. Therefore, this invention obtains the audio signal within a preset window based on the acoustic feature information, performs pre-emphasis, framing, and windowing processing on the audio signal to obtain a preprocessed audio frame, performs a Fast Fourier Transform on the preprocessed audio frame to obtain the spectrum, passes the spectrum through a Mel filter bank, and takes the logarithm of the filtered energy. Performing a discrete cosine transform to obtain Mel frequency cepstral coefficients is a standardization preprocessing and Mel frequency cepstral coefficient extraction step for the acoustic feature information of the live stream. Its physical significance lies in the fact that the original acoustic feature information is acquired in real time by the live stream audio acquisition module. It is a continuous time-domain audio signal with problems such as high-frequency attenuation and signal instability, making it impossible to directly extract features. However, the Mel frequency cepstral coefficient is a classic acoustic parameter that characterizes the audio spectrum features. It simulates the hearing characteristics of the human ear and can accurately reflect the core features of audio such as timbre and spectral distribution. By standardizing the preprocessing to transform the time-domain signal into a frequency-domain signal, and then extracting the parameter through the Mel filter bank and discrete cosine transform, the accurate quantization of the audio spectrum features can be achieved. In terms of technical implementation, the audio signal within a preset window is first extracted from the acoustic feature information obtained by the audio acquisition module. The window duration is set to 20-30ms, and the frame shift is 10ms. This parameter setting ensures both signal stability and the real-time performance of live acoustic detection. Then, the audio signal is pre-emphasized using a first-order high-pass filter with a coefficient of 0.97 to compensate for the attenuation of high-frequency components in the audio signal and improve the recognition of high-frequency features. Next, frame segmentation is performed to divide the continuous audio signal into multiple overlapping audio frames to solve the problem of processing non-stationary signals. Then, windowing is performed using a Hamming window to weight each audio frame to suppress spectral leakage. After the above processing, the pre-processed audio frame is obtained. The preprocessed audio frames are then subjected to a Fast Fourier Transform (FFT) to convert the time-domain audio frames into frequency-domain spectra, obtaining the amplitude and phase information at each frequency point. The spectrum is then passed through a Mel filter bank containing 24-40 triangular filters. The frequency scale of this filter bank closely matches the auditory characteristics of the human ear, providing higher resolution for low-frequency signals and effectively extracting audio features that the human ear is sensitive to. The energy of each filter bank after filtering is taken as the natural logarithm to compress the dynamic range. Finally, a Discrete Cosine Transform (DCT) is performed to remove the correlation between features in each dimension, resulting in Mel frequency cepstral coefficients with dimensions of 12-16. At the same time, a 1-dimensional frame energy feature can be added to form the final Mel frequency cepstral coefficient feature set.For example, in game live streaming scenarios, if a streamer uses abusive or offensive language, the spectral distribution of their audio will differ significantly from normal conversational speech. This difference is directly reflected in the characteristic values ​​of the Mel-frequency cepstral coefficients, which can accurately capture such spectral anomalies. The technical advantage of this step lies in solving problems such as high-frequency attenuation and spectral leakage in the original audio signal through a standardized audio preprocessing workflow, ensuring the accuracy of subsequent feature extraction. Furthermore, the extraction process of the Mel-frequency cepstral coefficients closely matches the characteristics of human hearing, and the resulting parameters accurately characterize the spectral features of the audio. At the same time, the reasonable setting of parameters such as window duration, frame shift, and the number of Mel filter banks ensures feature extraction accuracy while controlling computational power consumption, adapting to the application requirements of real-time live streaming detection and providing accurate spectral feature basis for subsequent acoustic violation judgment.

[0033] Based on the audio signal, candidate fundamental frequency values ​​are obtained, and a continuous fundamental frequency trajectory is extracted from the candidate fundamental frequency values ​​using a dynamic programming algorithm. The fundamental frequency trajectory is then smoothed to obtain the fundamental frequency parameters. This step involves extracting and optimizing the fundamental frequency parameters of the live stream audio signal. Its physical significance lies in the fact that the fundamental frequency is the basic frequency of the audio signal, directly representing the pitch and variation of the audio. Acoustic violations in live streams are often accompanied by abnormal increases, decreases, or fluctuations in the fundamental frequency. However, the directly extracted candidate fundamental frequency values ​​have many error points and discontinuities, making them unsuitable for direct violation judgment. By extracting a continuous fundamental frequency trajectory using a dynamic programming algorithm and smoothing it, accurate and continuous fundamental frequency parameters can be obtained, enabling the quantification of audio pitch characteristics. In terms of technical implementation, firstly, based on the preprocessed audio signal in step S31, the autocorrelation method is used to extract the fundamental frequency candidate values ​​of each audio frame. The autocorrelation method calculates the autocorrelation function of the audio signal, finds the delay value corresponding to the peak value of the function, and then calculates the fundamental frequency. This method has high computational efficiency and good noise resistance, and is suitable for the needs of real-time detection in live broadcasts. The extraction range of the fundamental frequency is set to 50-500Hz, covering the fundamental frequency range of human speech. Due to factors such as noise and the non-stationarity of audio signals, the extracted fundamental frequency candidate values ​​may contain erroneous values ​​and null values, resulting in discontinuous fundamental frequency trajectories. Subsequently, the following method is used... Dynamic programming is used to optimize the candidate fundamental frequency values. This algorithm defines state transition costs and local costs, and searches for a continuous trajectory with the minimum global cost among all candidate fundamental frequency values. It fills in missing values ​​and corrects erroneous values ​​to obtain a continuous fundamental frequency trajectory. Finally, a moving average method is used to smooth the continuous fundamental frequency trajectory. A sliding window of size 3-5 is selected, and neighborhood averaging is performed on each point of the fundamental frequency trajectory to remove local abnormal fluctuations, resulting in the final fundamental frequency parameter. This parameter is a continuous value that changes with the audio frame, accurately reflecting the real-time changes in audio pitch. For example, in entertainment live streaming scenarios, if the streamer uses inappropriate voices such as shouting or screaming, their fundamental frequency will suddenly and significantly increase, and the degree of fluctuation will increase significantly. These abnormal changes will be directly reflected in the fundamental frequency parameter, which can effectively capture such abnormal tonal characteristics. The technical benefits of this step are that the autocorrelation method ensures the efficiency and noise resistance of fundamental frequency candidate value extraction, the dynamic programming algorithm solves the problem of discontinuous fundamental frequency trajectory, and the smoothing process of the moving average method further improves the accuracy of fundamental frequency parameters. The obtained fundamental frequency parameters can accurately characterize the pitch change characteristics of audio, making up for the inadequacy of only extracting Mel frequency cepstral coefficients to characterize pitch features. This achieves comprehensive extraction of both audio spectrum and pitch features, providing a more comprehensive acoustic feature basis for subsequent audio violation detection models.

[0034] A pre-defined audio violation detection model is obtained, wherein the audio violation detection model is a binary classification neural network model trained based on a large number of labeled violation and non-violation audio samples. The Mel frequency cepstral coefficients and the fundamental frequency parameter are input into the audio violation detection model to obtain the abnormal audio probability value output by the audio violation detection model. The abnormal audio probability value is used as the abnormal audio component. This step involves inputting the extracted dual acoustic feature parameters into the pre-trained model to complete the violation probability determination, and finally obtaining the abnormal audio component in the acoustic dimension. Its physical significance is that the single Mel frequency cepstral coefficients and fundamental frequency parameter can only represent different features of the audio and cannot directly reflect the degree of violation of the audio. By using the pre-trained binary classification neural network model to fuse and detect the dual features, the feature parameters can be transformed into quantifiable violation probability values, thereby achieving an accurate representation of the degree of acoustic violation. In terms of technical implementation, a pre-defined audio violation detection model is first obtained. This model employs a shallow convolutional neural network (CNN) + fully connected layers design. The convolutional layers consist of two one-dimensional convolutional layers: the first layer has 32 kernels and a kernel size of 3, and the second layer has 64 kernels and a kernel size of 3, used to extract local features after fusing Mel-frequency cepstral coefficients and fundamental frequency parameters. The pooling layers use max pooling with a kernel size of 2, used for dimensionality reduction and preservation of key features. The fully connected layers consist of two layers with 128 and 64 neurons respectively. Finally, a Sigmoid algorithm is used. The id activation function outputs a probability value between 0 and 1. This shallow network structure ensures feature fusion and discrimination capabilities while reducing computational complexity, adapting to the computing power requirements of real-time live streaming detection. The training samples for this model consist of a large number of labeled live streaming violation and non-violation audio samples. Violation samples include types such as abusive language, shouting, and illegal sound effects, while non-violation samples include types such as normal communication, background music, and live streaming ambient sounds. The training process uses the cross-entropy loss function, the Adam optimizer, and a learning rate set to 0.001. Training continues until the model converges to ensure the model's discrimination accuracy. Subsequently, the Mel frequency cepstral coefficients obtained in step S31 and the fundamental frequency parameters obtained in step S32 are concatenated to form a fused acoustic feature vector, which is input into the audio violation discrimination model. After forward propagation, the model outputs an abnormal audio probability value, which is the abnormal audio component. The closer the value is to 1, the higher the degree of violation of the live streaming audio. For example, in e-commerce live streaming scenarios, if a host uses false advertising or other illegal language, the spectrum and pitch characteristics of their speech will differ from those of a normal explanation. After inputting the dual features of this speech into the model, the model will output a high probability value of abnormal audio, accurately representing the degree of acoustic violation.The technical advantage of this step is that the shallow neural network model design significantly reduces the inference time and computational power consumption of the model while ensuring the accuracy of discrimination, which can meet the needs of real-time detection in live broadcasts. The model trained on a large number of labeled samples has a strong ability to identify acoustic violation features. It can effectively integrate the dual feature information of Mel frequency cepstral coefficients and fundamental frequency parameters. The output abnormal audio probability value can accurately characterize the degree of violation in the acoustic dimension of live broadcasts. Moreover, this probability value is a scalar between 0 and 1, which forms a unified quantitative standard with the spatiotemporal violation component and the subsequent semantic violation component, providing convenience for the weighted fusion of subsequent multimodal violation components.

[0035] In one embodiment, step S4, which involves obtaining a text embedding vector as a semantic violation component based on the semantic feature information, and then performing a weighted fusion of the semantic violation component, the spatiotemporal violation component, and the anomalous audio component to obtain a comprehensive confidence score, includes: S41. Obtain real-time subtitle text or speech recognition text from the live stream based on the semantic feature information, and perform word segmentation and stop word removal on the real-time subtitle text or speech recognition text to obtain preprocessed text; S42. Obtain a pre-trained language representation model, input the pre-processed text into the pre-trained language representation model, obtain the text embedding vector output by the pre-trained language representation model, and use the text embedding vector as a semantic violation component. S43. Obtain a dynamic weight allocation strategy, wherein the dynamic weight allocation strategy is a mapping table that maps live streaming scene types to weight sets, and each weight set contains a semantic weight, a spatiotemporal weight and an audio weight. S44. Identify the scene type of the current live stream, and obtain the semantic weight corresponding to the semantic violation component, the spatiotemporal weight corresponding to the spatiotemporal violation component, and the audio weight corresponding to the abnormal audio component from the dynamic weight allocation strategy according to the scene type. The scene type identification adopts a lightweight classification model with multi-feature fusion. The training data of this model comes from the historical live stream annotation samples of the live streaming platform, including title text, cover image and multimodal features of the first 1 minute. The annotation method is manual annotation, and the model classification accuracy is not less than 95%. S45. Based on the semantic weight, the spatiotemporal weight, and the audio weight, the semantic violation component, the spatiotemporal violation component, and the abnormal audio component are weighted and summed to obtain a comprehensive confidence score. The semantic violation component is first converted into a scalar value between 0 and 1 through L2 normalization before participating in the weighted summation to ensure consistency of dimensions.

[0036] As described in steps S41-S45 above, since semantic violations in live streams are mainly reflected in the language text content, such as fraudulent rhetoric, abusive language, and illegal promotional statements, and the original semantic feature information is unstructured text content, it cannot directly participate in the quantitative calculation of the degree of violation. This invention obtains real-time subtitle text or speech recognition text from the live stream based on semantic feature information, and performs word segmentation and stop word removal on the real-time subtitle text or speech recognition text to obtain preprocessed text. This step is a standardized text preprocessing of the semantic feature information of the live stream. Its physical significance is that the original semantic text obtained by the live stream subtitle acquisition module and speech recognition module is unstructured data, containing a large number of stop words without actual semantic value, and the continuity of the text will affect the feature extraction effect of the subsequent model. Through word segmentation and stop word removal, invalid information can be eliminated and the text structure can be standardized, laying the foundation for the quantitative extraction of subsequent semantic features. In terms of technical implementation, semantic feature information is acquired in two ways. For live streams with real-time subtitles, the real-time subtitle text is directly extracted through the subtitle acquisition module. For live streams without real-time subtitles, the speech signal in the acoustic feature information of the live stream is converted into text through a speech recognition model. The speech recognition model adopts a lightweight end-to-end ASR model to ensure the real-time performance and accuracy of the transcription. Subsequently, the acquired original text is processed by word segmentation using a word segmentation algorithm based on bidirectional maximum matching. This algorithm combines the results of forward and reverse maximum matching to effectively solve the problem of ambiguous word segmentation, dividing the continuous text into independent lexical units. The word segmentation dictionary adopts a dictionary specifically for the live streaming field, which includes commonly used words in live streaming, popular internet slang, etc., to improve the accuracy of word segmentation. Then, stop word removal is performed. The stop word list includes words with no semantic value such as punctuation marks, modal particles, prepositions, and conjunctions. At the same time, meaningless commonly used words in the live streaming field are added. By traversing the word segmentation results and removing stop words, the preprocessed text composed of effective words is finally obtained. For example, in e-commerce live streaming, the original text might contain "Dear customers, this product is truly the lowest price online, you'll really regret it if you don't buy it!" After word segmentation, it becomes "Dear customers, this product is truly the lowest price online, you'll really regret it if you don't buy it!" After stop word removal, it becomes "This product is the lowest price online, you'll regret it if you don't buy it!" This process removes words with no semantic value while retaining the core semantic content. The technical advantage of this step lies in the fact that a standardized text preprocessing workflow removes invalid information from the original text, standardizes the text structure, and avoids meaningless words interfering with subsequent semantic feature extraction. Furthermore, the use of a dedicated word segmentation dictionary and stop word list for live streaming makes the preprocessing results more closely match the text characteristics of the live streaming scenario, improving the accuracy of subsequent semantic feature extraction. The lightweight speech recognition and word segmentation algorithms, while ensuring processing effectiveness, control computational consumption, adapting to the application requirements of real-time live streaming detection.

[0037] A pre-trained language representation model is obtained by inputting the pre-processed text into the pre-trained language representation model to obtain the text embedding vector output by the pre-trained language representation model. The text embedding vector is used as a semantic violation component. This step transforms the standardized pre-processed text into a quantifiable text embedding vector, realizing the quantitative extraction of semantic violation features. Its physical significance is that the pre-processed text is still a discrete lexical unit and cannot directly participate in the quantitative calculation of the degree of violation. However, the pre-trained language representation model can transform discrete text into a continuous high-dimensional numerical vector. This vector can accurately represent the semantic features of the text. Texts with different semantics will correspond to different vector distributions. Text vectors with violation semantics will have obvious differences from normal semantics. Using this vector as a semantic violation component can realize the quantitative representation of semantic violation features. In terms of technical implementation, the pre-trained language representation model adopts a lightweight BERT-base model, which is fine-tuned for the text characteristics of the live streaming domain. The fine-tuning samples are a large number of labeled live streaming violation and non-violation texts. The fine-tuning process adopts a joint training method of masked language model and next sentence prediction, with a learning rate set to 2e-5 and a training batch size of 32. Iteration is performed until the model converges, allowing the model to better capture the semantic features of the live streaming domain. The network structure of this model contains a 12-layer Transformer encoder, each layer containing a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism has 12 heads, which can capture the semantic association between words from different dimensions. The hidden layer dimension of the feedforward neural network is 768, and the final output text embedding vector has a dimension of 768. Each value in the vector is a quantized value representing the semantic features of the text. After the preprocessed text obtained in step S41 is vectorized and encoded, it is input into the fine-tuned pre-trained language representation model. After forward propagation, the model obtains the semantic feature vector of the entire text, i.e., the text embedding vector, through the output layer corresponding to the token. This vector is directly used as the semantic violation component. For example, if a live stream contains inappropriate text such as "Add me on WeChat to receive benefits, private transactions are more cost-effective," the resulting text embedding vector, after preprocessing and input into the model, will show a significant difference from the vector of normal live stream communication text. This vector can accurately represent the semantic features of the inappropriate text. The technical effect of this step is that the lightweight BERT-base model, fine-tuned for the live streaming domain, retains the powerful semantic feature extraction capabilities of the pre-trained language model while improving the semantic representation accuracy and inference speed of the model in live streaming scenarios through lightweight design and domain fine-tuning. The output 768-dimensional text embedding vector can accurately and comprehensively represent the semantic features of live stream text, transforming unstructured semantic text into quantifiable numerical vectors, achieving effective extraction of semantic violation components, and providing a standardized quantitative basis for the subsequent fusion of multimodal violation components. At the same time, the model's inference speed can meet the computational power requirements of real-time live streaming detection.

[0038] The dynamic weight allocation strategy is obtained, which is a mapping table that maps live streaming scene types to weight sets. Each weight set contains a semantic weight, a spatiotemporal weight, and an audio weight. This step is to build a scenario-based weight allocation basis for the subsequent weighted fusion of multimodal violation components. Its physical significance is that the content characteristics of different live streaming scenes determine that the importance of the violation features in the three dimensions of visual, acoustic, and semantics to the overall violation judgment is different. A single fixed weight cannot adapt to the detection needs of all scenarios. By constructing a mapping table between live streaming scene types and weight sets, corresponding differentiated weights can be matched for different scenarios, making the results of multimodal fusion more in line with the actual violation detection needs of each scenario. In terms of technical implementation, the dynamic weight allocation strategy is stored in the form of a mapping table. The row dimension of the mapping table represents the live streaming scenario type, covering mainstream live streaming scenarios such as e-commerce live streaming, game live streaming, entertainment live streaming, outdoor live streaming, and knowledge live streaming. The column dimension represents the weight type, including semantic weight, spatiotemporal weight, and audio weight. Each scenario type corresponds to a weight set, and the three weight values ​​in the weight set are all values ​​between 0 and 1, and the sum of the three weight values ​​is 1, ensuring the consistency of the dimensions of the weighted fusion result. The numerical setting of the weight set is based on the analysis of violation features and experimental verification of labeled samples from a large number of live streaming scenarios, targeting semantic violations. For e-commerce live streams, which have the highest rate of violations, a semantic weight of 0.6, a spatiotemporal weight of 0.2, and an audio weight of 0.2 are assigned. For game live streams, which have the highest rate of acoustic violations, an audio weight of 0.5, a semantic weight of 0.3, and a spatiotemporal weight of 0.2 are assigned. For entertainment live streams, which have the highest rate of visual violations, a spatiotemporal weight of 0.5, a semantic weight of 0.3, and an audio weight of 0.2 are assigned. For outdoor live streams and knowledge live streams, which have a relatively balanced distribution of violations across all dimensions, all three weight values ​​are set to around 0.33. The mapping table can be flexibly adjusted according to the violation detection needs of the live streaming platform, exhibiting good scalability. The technical effect of this step is that by constructing a scenario-based weight mapping table, a scientific and flexible allocation basis is provided for the weighted fusion of multimodal violation components. This breaks the limitations of traditional fixed weights, allowing weight allocation to fit the content characteristics and violation detection focus of different live streaming scenarios. Furthermore, the storage format and adjustability of the mapping table allow this strategy to quickly adapt to changes in live streaming scenarios and adjustments in platform detection needs, laying the foundation for subsequent accurate calculation of comprehensive confidence scores.

[0039] The process involves identifying the scene type of the current live stream and obtaining the semantic weights corresponding to the semantic violation components, the spatiotemporal weights corresponding to the spatiotemporal violation components, and the audio weights corresponding to the abnormal audio components from the dynamic weight allocation strategy based on the scene type. This step completes the identification of the current live stream scene type and the matching of the corresponding weights. Its physical significance lies in the fact that the dynamic weight allocation strategy is a preset mapping table. It is necessary to accurately identify the actual scene type of the current live stream before matching the corresponding weight set from the mapping table to achieve scenario-based weighted fusion of multimodal violation components. The accuracy of scene identification directly determines the rationality of weight matching, which in turn affects the accuracy of the comprehensive confidence score. In terms of technical implementation, the live stream scene type identification adopts a lightweight classification model with multi-feature fusion. The input features of the model include the title text features, cover image features, and initial 1-minute multimodal features of the live stream. The title text features are extracted through the pre-trained language representation model in step S42, the cover image features are extracted through the lightweight ResNet18 model, and the initial 1-minute multimodal features are fused features of spatiotemporal violation components, abnormal audio components, and semantic violation components. The network structure of the classification model is a shallow fully connected network, containing two fully connected layers with 256 and 128 neurons, respectively. Finally, the probability values ​​of each scene type are output through the Softmax activation function, and the scene type with the highest probability value is selected as the scene type of the current live stream. The training data of this model comes from the historical live stream labeled samples of the live streaming platform, including title text, cover image, and initial 1-minute multimodal features. The labeling method is manual labeling, and the model classification accuracy is not less than 95%. After completing scene identification, the corresponding semantic weights, spatiotemporal weights, and audio weights are retrieved from the mapping table of the dynamic weight allocation strategy using the scene type as an index, as the basis for subsequent weighted fusion calculations. For example, if the current live stream is identified as an e-commerce live stream, then a semantic weight of 0.6, a spatiotemporal weight of 0.2, and an audio weight of 0.2 are retrieved from the mapping table. If it is identified as a game live stream, then an audio weight of 0.5, a semantic weight of 0.3, and a spatiotemporal weight of 0.2 are retrieved. The technical effect of this step is that the lightweight classification model based on multi-feature fusion can accurately and quickly identify the scene type of the live stream, balancing recognition accuracy and real-time performance, avoiding scene misjudgment caused by single feature recognition. Furthermore, the scene type-based weight indexing method can quickly and accurately match the corresponding weight set from the dynamic weight allocation strategy, providing accurate calculation parameters for subsequent weighted fusion and ensuring the scene adaptability of multimodal fusion.

[0040] Based on the semantic weight, the spatiotemporal weight, and the audio weight, the semantic violation component, the spatiotemporal violation component, and the abnormal audio component are weighted and summed to obtain a comprehensive confidence score. This step completes the contextualized weighted fusion of multimodal violation components, ultimately obtaining a comprehensive confidence score that represents the overall violation level of the live stream. Its physical significance lies in the fact that the semantic violation component, the spatiotemporal violation component, and the abnormal audio component quantify the violation level of the live stream from three dimensions. By weighting and summing the three components with differentiated weights matched with the scenario, the violation features of the three dimensions can be integrated into a comprehensive quantitative indicator. This indicator can accurately reflect the overall violation level of the live stream, providing a clear and quantifiable judgment standard for the subsequent screening of low-confidence samples. In terms of technical implementation, the semantic violation component is first normalized. Since the semantic violation component is a high-dimensional vector of 768 dimensions, it is first converted into a scalar value between 0 and 1 through L2 normalization. The spatiotemporal violation component and the abnormal audio component are themselves scalar values ​​between 0 and 1, and do not require additional normalization. Then, a weighted sum is performed according to the formula: the comprehensive confidence score = semantic violation component scalar value × semantic weight + spatiotemporal violation component × spatiotemporal weight + abnormal audio component × audio weight. The calculated comprehensive confidence score is a scalar value between 0 and 1. The closer the value is to 1, the higher the overall violation level of the live stream. The closer the value is to 0, the higher the compliance level of the live stream. For example, if the semantic violation component of an e-commerce live stream has a scalar value of 0.8, the spatiotemporal violation component has a value of 0.3, and the abnormal audio component has a value of 0.2, and the matching weights are semantic 0.6, spatiotemporal 0.2, and audio 0.2, then the overall confidence score = 0.8 × 0.6 + 0.3 × 0.2 + 0.2 × 0.2 = 0.58. This value can comprehensively reflect the overall violation level of the e-commerce live stream. Moreover, because the semantic weight has the highest proportion, the semantic violation component contributes the most to the result, which is in line with the detection characteristics of e-commerce live streams. The technical effect of this step is that L2 normalization unifies the dimensions of high-dimensional semantic violation components with other scalar violation components, ensuring the rationality of weighted fusion. The weighted summation algorithm based on scene weights has low computational complexity and fast inference speed, which can meet the needs of real-time detection in live streaming. The final scalar comprehensive confidence score between 0 and 1 can accurately and comprehensively represent the overall violation degree of the live stream, and set a clear and quantifiable judgment standard for the subsequent screening of low confidence samples. At the same time, the calculation of this score is in line with the detection focus of the current live streaming scenario, improving the accuracy and scene adaptability of the overall violation judgment.

[0041] In one embodiment, step S6, which involves performing few-sample learning on the low-confidence samples to obtain a small-sample adaptation vector classification head, and incrementally updating the multimodal feature extractor based on the adaptation vector classification head to obtain the incremental update result, includes: S61. Obtain the multimodal data of samples labeled as low-confidence samples, and use the multimodal data of the low-confidence samples as a support set; S62. Input the support set into the multimodal feature extractor to obtain the feature vector of each sample in the support set, wherein the multimodal feature extractor is the version before incremental update, used to extract features of the support set samples; S63. Obtain the violation type label for each sample in the support set, and group the feature vectors according to the violation type label, and calculate the prototype vector of each group of feature vectors; S64. Construct an adaptation vector classification head based on the prototype vector, wherein the adaptation vector classification head is used to calculate the distance between the feature vector of the sample to be classified and the prototype vector. The distance is calculated using "1-cosine similarity", with a value range of 0-1. The smaller the value, the more similar the sample. S65. Obtain a preset incremental update strategy, integrate the adaptation vector classification head into the output of the multimodal feature extractor according to the incremental update strategy, and only fine-tune the parameters of the last two layers of the multimodal feature extractor, while fixing the parameters of the remaining layers, to obtain the incremental update result.

[0042] As described in steps S61-S65 above, this invention acquires multimodal data labeled as low-confidence samples and uses the multimodal data of the low-confidence samples as a support set. This step lays the data foundation for few-shot learning. Its physical significance is that low-confidence samples are samples that the model cannot accurately determine. These samples contain novel or ambiguous violation features. Integrating them into a support set can provide targeted training data for subsequent few-shot learning, allowing the model to learn the representation rules of novel violation features with a small number of samples without the need for a large number of labeled samples. In terms of technical implementation, the low-confidence sample multimodal data refers to the visual, acoustic, and semantic multimodal data of the live stream obtained in step S1 and determined by step S5 to have a comprehensive confidence score below a preset threshold. This data is automatically marked by the system and stored in a dedicated sample library. During incremental updates, the marked multimodal data of this type are retrieved in batches from the sample library, initially classified according to violation type, and integrated into a small sample support set. The sample size of the support set is set according to the complexity of the new violation features, generally 10-50 samples per violation type, which meets the sample size requirements for small sample learning. At the same time, the support set retains the complete multimodal feature information of the samples and the violation type labels after manual review, ensuring the accuracy of subsequent feature extraction and classification head construction. For example, when a new type of disguised violation language appears in the live stream, the original model cannot accurately determine it. These samples will be marked as low-confidence samples. After being integrated into the support set, they can provide data support for the model to learn this type of new semantic violation feature. The technical advantage of this step is that by constructing a small support set by screening low-confidence samples, the incremental updates of the model have a clear focus. The model learns only new violation features that the original model cannot identify, thus avoiding meaningless model updates. At the same time, the small sample size significantly reduces the cost of sample labeling and data processing, which is suitable for the actual situation in live streaming scenarios where new violation patterns emerge rapidly and it is difficult to quickly obtain a large number of labeled samples.

[0043] The support set is input into the multimodal feature extractor to obtain the feature vector of each sample in the support set. The multimodal feature extractor is the version before incremental update, used to extract features from the support set samples. This step is to extract multimodal features from the support set samples, transforming the original multimodal data into a computable feature vector. Its physical significance is that the original multimodal data in the support set is the original feature information of vision, acoustics, and semantics, which cannot be directly used for the calculation of prototype vectors and the construction of classification heads. By extracting and fusing features through the multimodal feature extractor, a high-dimensional feature vector integrating multi-dimensional features can be obtained. This vector can accurately represent the overall features of the sample, laying the foundation for subsequent analysis and calculation based on feature vectors. In terms of technical implementation, the multimodal feature extractor is a pre-defined multimodal network model consisting of a visual feature extraction sub-network, an acoustic feature extraction sub-network, a semantic feature extraction sub-network, and a feature fusion layer. The visual feature extraction sub-network uses the lightweight MobileNetV3 model to extract the spatiotemporal features of visual feature information. The acoustic feature extraction sub-network uses a one-dimensional convolutional neural network to extract the spectral and pitch features of acoustic feature information. The semantic feature extraction sub-network uses the lightweight BERT-base model mentioned in S42 above to extract the text embedding features of semantic feature information. The feature fusion layer uses a concatenation fusion method to concatenate the feature vectors extracted by the three sub-networks into a high-dimensional comprehensive feature vector. The multimodal data of each sample in the support set is input into the corresponding feature extraction sub-network. After each sub-network extracts a single-modal feature vector, the feature fusion layer completes the multimodal feature fusion, finally obtaining a high-dimensional comprehensive feature vector for each sample. The dimension of this vector is set according to the output feature dimension of each sub-network, generally 1024 dimensions, which can comprehensively represent the multimodal features of the sample. The technical effect of this step is that it achieves effective extraction and fusion of multi-dimensional features of support set samples through a multimodal feature extractor. The resulting high-dimensional feature vector can accurately and comprehensively represent the overall features of the sample, providing a standardized quantitative basis for the subsequent calculation of prototype vectors. At the same time, the multimodal feature extractor adopts a lightweight sub-network structure, which ensures the speed of feature extraction, adapts to the computing power requirements of live real-time detection, and the fixed feature extraction method allows the feature vectors of different samples to have a unified dimension, which is convenient for subsequent clustering and calculation.

[0044] The violation type label of each sample in the support set is obtained, and the feature vectors are grouped according to the violation type label. The prototype vector of each group of feature vectors is calculated. This step is to perform cluster analysis on the feature vectors of the support set samples and calculate the prototype vector. Its physical meaning is that the feature vectors of samples of the same violation type will show similar distribution patterns. By grouping the feature vectors by violation type label and then calculating the prototype vector of each group, the core representation vector of this type of violation feature can be obtained. The prototype vector can represent the overall feature pattern of this new type of violation feature, providing a core basis for the subsequent construction of the adaptation vector classification head. In terms of technical implementation, the system supports the use of precise labels for the violation type of each sample, manually reviewed and annotated, covering all violation types defined by the live streaming platform, including traditional violation types and newly emerging variant violation types. First, based on the violation type label, all sample feature vectors obtained in step S62 are grouped. Feature vectors of the same violation type are grouped together, while feature vectors of different violation types are grouped into different groups. Then, a prototype vector is calculated for the feature vectors within each group. The prototype vector is calculated using a simple arithmetic mean method, that is, the average value of the corresponding dimension values ​​of all feature vectors within each group is calculated, and the resulting average vector is the prototype vector for that group. This calculation method is simple and efficient, requiring no complex clustering algorithms, and can quickly obtain the prototype vector. Furthermore, the arithmetic mean method can effectively represent the central distribution of a group of feature vectors, accurately reflecting the core pattern of this type of violation feature. The dimension of the prototype vector is consistent with the dimension of the sample feature vectors, ensuring consistency in subsequent calculations. For example, the sample feature vectors supporting all new types of disguised violation language are grouped together, and the average value of all 1024-dimensional feature vectors within that group is calculated dimension by dimension. The resulting 1024-dimensional average vector is the prototype vector for this type of new semantic violation. The technical effect of this step is that it achieves precise grouping of feature vectors through violation type labels, giving the calculation of prototype vectors a clear category orientation. The use of the arithmetic mean method, while ensuring that the prototype vectors can accurately represent the core features of this type of violation, significantly reduces the computational complexity and improves the calculation speed of prototype vectors, adapting to the needs of rapid incremental updates of the model. The obtained prototype vectors serve as the core representation of this type of violation feature, providing a key basis for the subsequent construction of the adaptive vector classification head.

[0045] An adaptation vector classification head is constructed based on the prototype vector. This adaptation vector classification head calculates the distance between the feature vector of the sample to be classified and the prototype vector. The distance is calculated using "1-cosine similarity," with a value ranging from 0 to 1. A smaller value indicates greater similarity. This step involves constructing an adaptation vector classification head specifically for identifying novel violation features based on the prototype vector. Physically, the prototype vector represents the core representation of various novel violation features. Constructing an adaptation vector classification head with the prototype vector as its core allows for the determination of whether a sample belongs to a certain type of violation by calculating the distance between the feature vector of the sample to be classified and the prototype vector, thus achieving accurate classification of novel violation patterns. This classification head, as a supplement to the original model, enables the updated model to recognize novel violation features. In terms of technical implementation, the adaptation vector classification head is a lightweight classification module based on distance metrics. Its core is a prototype vector library of various violation features, and it includes a distance calculation unit and a classification decision unit. The distance calculation unit uses the "1-cosine similarity" algorithm to calculate the cosine distance between the feature vector of the sample to be classified and each prototype vector in the prototype vector library. The cosine distance ranges from 0 to 1; a smaller value indicates a higher similarity between the sample's feature vector and the prototype vector, while a larger value indicates a lower similarity. The classification decision unit presets a distance threshold, which is based on the "1-cosine similarity" between the feature vector of the support set sample and its corresponding prototype vector. The "degree" statistical result is typically set to 0.3. When the cosine distance between the feature vector of the sample to be classified and a certain prototype vector is less than this threshold, the sample is determined to belong to this type of violation. If all cosine distances are greater than this threshold, the sample is determined to be non-violation or an unidentified violation type. The network structure of the adaptive vector classification head is a lightweight structure without training parameters, containing only basic modules for vector storage and distance calculation. It does not require complex convolutional or fully connected layers, enabling it to quickly complete distance calculation and classification. Furthermore, this classification head is flexibly expandable; when a new violation type appears, only the corresponding prototype vector needs to be added to the prototype vector library, without reconstructing the classification head. The technical effect of this step is that the adaptive vector classification head built based on prototype vectors can accurately identify new violation samples with features similar to the prototype vectors, realizing the model's ability to judge new violation patterns. At the same time, the classification head adopts a lightweight distance metric structure without complex network layers and training parameters, which reduces computational consumption and improves the speed of classification, adapting to the needs of live real-time detection. Moreover, the scalability of the classification head allows the model to continuously adapt to constantly emerging new violation patterns, extending the model's lifespan.

[0046] A preset incremental update strategy is obtained. Based on this strategy, the adaptive vector classification head is integrated into the output of the multimodal feature extractor. Only the parameters of the last two layers of the multimodal feature extractor are fine-tuned, while the parameters of the remaining layers are fixed, resulting in an incremental update. This step completes the integration of the adaptive vector classification head and the fine-tuning of the local parameters of the multimodal feature extractor, achieving lightweight incremental updates of the model. Physically, the adaptive vector classification head, being a newly added classification module, needs to be integrated with the original multimodal feature extractor to form a complete updated model. Fine-tuning only the parameters of the last two layers of the multimodal feature extractor allows the original model's feature extraction capabilities to better adapt to the representation of new illegal features. Simultaneously, fixing the parameters of the remaining layers effectively prevents the model from forgetting its ability to identify existing illegal features, avoiding catastrophic forgetting and achieving smooth incremental updates of the model. In terms of technical implementation, the preset incremental update strategy is a model update rule pre-defined by the platform, which clarifies the integration method of the classification head, the range of network layers for parameter fine-tuning, and the hyperparameter settings for fine-tuning. First, following this strategy, the adapted vector classification head constructed in step S64 is integrated into the output of the multimodal feature extractor, forming a parallel dual-classification head structure with the original classification head. The feature vector output by the multimodal feature extractor can be simultaneously input to both classification heads for independent judgment. Then, parameter fine-tuning is performed, with the fine-tuning scope limited to the last two layers of the multimodal feature extractor, namely the feature fusion layer and the previous layer. The parameters of the mapping layer and all other network layers remain fixed. The fine-tuned hyperparameters are set as follows: using the Adam optimizer, the learning rate is set to 1e-5, which is much lower than the learning rate of full retraining; the training batch size is 8; the training epochs are 10-20; and the training loss function is contrastive loss, with the similarity between the feature vectors of the support set samples and the corresponding prototype vectors as the optimization objective. This allows the feature vectors extracted by the multimodal feature extractor to more accurately represent novel violation features. After fine-tuning, the multimodal feature extractor, which integrates the adaptive vector classification head and completes local parameter fine-tuning, becomes the final incremental update result. For example, after learning the support set containing novel disguised violation language, the adaptive vector classification head is integrated into the output of the original model, and only the parameters of the last two layers of the multimodal feature extractor are fine-tuned. This allows the model to more accurately extract the features of this type of novel semantic violation, while retaining the original model's ability to identify traditional semantic violations such as insults and fraud.The technical advantage of this step lies in the parallel integration of the dual-classification heads, which allows the updated model to simultaneously identify both traditional and novel violation patterns, thus improving the overall detection range of the model. Furthermore, the method of fine-tuning only the parameters of the last two network layers significantly reduces the computational cost and time cost of model updates. Compared to full retraining, computational cost can be reduced by more than 80%, and the update cycle can be shortened to a few minutes, enabling rapid adaptation to the emergence of new violation patterns in live streaming scenarios. At the same time, the fine-tuning settings with low learning rate and few training rounds, combined with the fixed parameters of the remaining network layers, effectively prevent catastrophic forgetting of the model, ensuring that the model does not lose the recognition accuracy of the original violation features while learning new violation features, thus achieving stable and efficient incremental updates of the model.

[0047] In one embodiment, step S6, which provides multi-type violation warnings for the live stream based on the incremental update result, includes: S66. Obtain the incremental update result, wherein the incremental update result includes the updated multimodal feature extractor and the newly added adaptation vector classification head; S67. Obtain the feature vector of the new multimodal data in the live stream according to the updated multimodal feature extractor; S68. Obtain the corresponding preset original classification head based on the multimodal data, and input the feature vector into the preset original classification head and the newly added adaptation vector classification head respectively to obtain the original classification result and the adaptation classification result; S69. Obtain the decision fusion rule, wherein the decision fusion rule is: if the original classification result and the adapted classification result are consistent, then the original classification result is taken as the final classification result; if they are inconsistent, then the result with higher confidence is selected as the final classification result, and the original classification result and the adapted classification result are fused according to the decision fusion rule to obtain the final classification result; S610. Determine whether there is any violation in the live stream based on the final classification result. If the final classification result exceeds a preset violation threshold, generate a warning message and issue multiple types of violation warnings to the live stream based on the warning message.

[0048] As described in steps S66-S610 above, the present invention obtains the incremental update result, wherein the incremental update result includes the updated multimodal feature extractor and the newly added adaptive vector classification head. This step is the basic preparatory step for the entire incremental update violation warning process. Its physical significance is that the incremental update result is the optimized core component of the model. The updated multimodal feature extractor has improved its ability to extract new violation features, and the newly added adaptive vector classification head has the ability to determine new violation patterns. Only by accurately obtaining the incremental update result can a qualified model foundation be provided for the feature extraction and dual classification head determination of subsequent new data, ensuring the effectiveness of the subsequent warning process. In terms of technical implementation, the incremental update results are uniformly stored and scheduled by the system's model management module. This module encrypts and stores the multimodal feature extractor parameters, the prototype vector library of the adapted vector classification head, and the distance judgment parameters that have completed the incremental update, and generates a unique model version identifier. When executing the violation warning process, the system retrieves the latest incremental update results from the model management module based on the real-time needs of live detection, using the version identifier. Simultaneously, it loads the original preset classification head to ensure the integrity of the dual-classification head architecture. The retrieval process uses a lightweight model parameter transmission protocol to ensure fast model loading and adapt to the needs of real-time live detection. The obtained incremental update results maintain interface consistency with the original model, requiring no additional modifications to the code of subsequent detection processes, achieving seamless model switching. The technical effect of this step is that, through the unified scheduling and seamless loading of the model management module, incremental update results are quickly and accurately obtained, providing an optimized model foundation for subsequent feature extraction and dual-classification head judgment. At the same time, the interface consistency design ensures a smooth transition in the detection process after the model update, avoiding detection interruptions caused by model updates and guaranteeing the real-time and continuous nature of live violation warnings.

[0049] The updated multimodal feature extractor is used to obtain feature vectors of new multimodal data in the live stream. This step involves multimodal feature extraction of the new live stream data, transforming the original live stream data into a quantified feature vector that can be used for classification head determination. The physical significance of this step is that the new multimodal data in the live stream consists of real-time collected visual, acoustic, and semantic raw feature information, which cannot be directly input into the classification head for violation determination. The updated multimodal feature extractor, after local parameter fine-tuning, has optimized the extraction capabilities of both traditional and new violation features. By using this extractor to extract and fuse features from the new data, a more accurate high-dimensional comprehensive feature vector can be obtained, providing a reliable quantitative basis for the accurate determination of the subsequent dual classification head. In terms of technical implementation, the new multimodal data in the live stream is acquired by the system's real-time acquisition module according to the acquisition rules of step S1. This data includes visual, acoustic, and semantic features. The acquisition frequency matches the bitrate of the live stream to ensure the real-time performance and completeness of the feature information. The newly acquired multimodal data is then input into the visual, acoustic, and semantic feature extraction sub-networks of the updated multimodal feature extractor according to their modality types. Each sub-network extracts a single-modality feature vector according to preset feature extraction rules. The visual sub-network still uses the MobileNetV3 model to extract spatiotemporal features, the acoustic sub-network uses a one-dimensional convolutional neural network to extract spectral and pitch features, and the semantic sub-network uses a lightweight BERT-base model to extract text embedding features. The parameters of each sub-network have been fine-tuned through incremental updates, resulting in higher accuracy in extracting new violation features. Subsequently, the three single-modal feature vectors are concatenated and fused through a feature fusion layer to generate a high-dimensional comprehensive feature vector with a dimension of 1024. This vector integrates the multi-dimensional features of the new data and can accurately represent both traditional and new violation features simultaneously. For example, for new live stream data containing novel disguised violation phrases, the updated semantic feature extraction subnetwork can more accurately extract the semantic features of the phrases. The fused comprehensive feature vector will clearly show the representational pattern of the novel violation feature, providing an accurate basis for subsequent classification head judgment. The technical effect of this step is that the updated multimodal feature extractor, with its fine-tuned parameters, achieves accurate extraction and fusion of traditional and novel violation features in the new live stream data. The resulting comprehensive feature vector can comprehensively and accurately represent the multimodal features of the new data, providing a high-quality quantitative basis for the parallel judgment of the subsequent dual classification head. At the same time, the feature extraction process and speed remain unchanged, ensuring the computing power and time requirements of real-time live stream detection, and the real-time performance of detection is not reduced due to incremental model updates.

[0050] Based on the multimodal data, a corresponding preset original classification head is obtained. The feature vector is then input into both the preset original classification head and the newly added adaptive vector classification head to obtain the original classification result and the adapted classification result. This step involves parallel judgment using two classification heads on the extracted feature vector to obtain two independent violation classification results. The physical significance of this step is that the preset original classification head is trained on a large number of traditional violation samples, giving it high confidence in judging traditional violation patterns. The newly added adaptive vector classification head is constructed based on small sample learning from low-confidence samples, giving it a unique advantage in judging novel violation patterns. By simultaneously inputting the same feature vector into two classification heads for parallel and independent judgment, the respective advantages of the two classification heads can be fully utilized to obtain two classification results from different perspectives, providing rich judgment basis for subsequent result fusion and avoiding the limitations of a single classification head. In terms of technical implementation, the preset original classification head is the core classification module before the model's incremental update. It is a multi-class neural network model trained on a large number of traditional live streaming violation and non-violation samples. The network structure contains two fully connected layers with 512 and 64 neurons respectively. Finally, the Softmax activation function is used to output the confidence value of each violation type, with the confidence value ranging from 0 to 1, and the result of determining the violation type or non-violation. The newly added adaptive vector classification head is a distance metric classification module constructed through step S64. Its core consists of a prototype vector library and a 1-cosine similarity calculation unit. It calculates the similarity between the feature vector and the prototype vector library. The 1-cosine similarity of each prototype vector is used to output the violation type or non-violation result, along with the confidence value of the judgment result. The confidence value is obtained by subtracting the minimum 1-cosine similarity from 1, and the range is also 0-1, ensuring consistency with the confidence dimension of the original classification result. The comprehensive feature vector obtained in step S612 is simultaneously input into the preset original classification head and the newly added adaptive vector classification head. The two classification heads perform parallel and independent inference calculations without interfering with each other. After the inference is completed, the original classification result and the adaptive classification result are output respectively. Each classification result includes the judgment result type (a certain type of violation / non-violation) and the corresponding confidence value. For example, for a feature vector containing a new type of disguised violation, the original classification head may output a non-violation result due to insufficient learning of the feature, with a confidence value of 0.6, while the adaptive vector classification head can accurately identify the feature and output the corresponding new semantic violation result, with a confidence value of 0.9. The two classification heads form complementary judgment results.The technical advantage of this step lies in the fact that the parallel judgment method of the dual classification head fully leverages the advantages of the original classification head in judging traditional violations and the advantages of the adapted classification head in judging new violations. The two independent classification results obtained provide a more comprehensive judgment basis for subsequent fusion, avoiding the omission or misjudgment of some violation patterns by a single classification head. At the same time, the parallel inference calculation method does not increase the overall judgment time, ensuring the real-time performance of live detection. The unified confidence value output provides a standardized calculation basis for subsequent result fusion.

[0051] The decision fusion rule is as follows: if the original classification result and the adapted classification result are consistent, then the original classification result is taken as the final classification result; if they are inconsistent, then the result with higher confidence is selected as the final classification result, and the original classification result and the adapted classification result are fused according to the decision fusion rule to obtain the final classification result. This step fuses the judgment results of the two classification heads according to scientific rules to obtain a final classification result that is both comprehensive and accurate. Its physical significance is that the judgment results of the two classification heads may be consistent or inconsistent. When they are consistent, it means that the two classification heads have reached a consensus on the judgment of the sample, and the credibility of the result is higher. When they are inconsistent, it means that one classification head has a greater advantage in judging the features of the sample. In this case, selecting the result according to the confidence level can ensure the accuracy of the final classification result. Through this decision fusion rule, the judgment results of the two classification heads can be intelligently integrated, taking the best and complementing the shortcomings, to obtain a more accurate final classification result than a single classification head. In terms of technical implementation, the decision fusion rule is pre-set and stored by the system's rule management module. This rule is a deterministic logical judgment rule without complex calculation processes, facilitating rapid system execution. First, the decision fusion rule is retrieved from the rule management module. Then, the system's result fusion module extracts the result type and confidence value from the original classification result and the adapted classification result obtained in step S613. It first performs a consistency judgment on the result type. If the result types determined by the two are the same, regardless of whether it is a specific violation type or not a violation, the result type and the average of the two confidence values ​​are directly used as the final classification result. If the result types determined by the two are different, the confidence values ​​of the two results are compared, and the result type with the higher confidence value and its corresponding confidence value are selected as the final classification result. In the extreme case where the confidence values ​​are equal, the adapted classification result is selected by default to prioritize the identification of new violation patterns. The entire result fusion process is completed by the system's lightweight logic computing unit, with a computation time in the microsecond range, which will not affect the real-time performance of live detection. For example, when the original classification result is non-violation (confidence 0.6) and the adapted classification result is novel semantic violation (confidence 0.9), the final classification result is novel semantic violation (confidence 0.9) because the result types are inconsistent and the adapted classification result has a higher confidence. When both classification results are abusive violations with confidences of 0.8 and 0.85 respectively, the final classification result is abusive violation with a confidence of 0.825.The technical advantage of this step lies in the fact that the simple and efficient decision fusion rules enable the intelligent integration of the judgment results from the dual-classification head, effectively solving the limitations of the judgment of a single classification head. This allows the final classification result to retain the original classification head's accurate judgment capability for traditional violations while incorporating the adaptive classification head's exclusive judgment capability for new types of violations, significantly improving the comprehensiveness and accuracy of the violation classification results. At the same time, the lightweight logical calculations allow the result fusion process to be completed quickly, ensuring the needs of real-time detection during live streaming. The determinism of the rules also makes the fusion results interpretable, facilitating subsequent early warning analysis and model optimization.

[0052] Based on the final classification result, it is determined whether there are any violations in the live stream. If the final classification result exceeds a preset violation threshold, a warning message is generated, and multiple types of violation warnings are issued to the live stream based on the warning message. This step is the final execution stage of the entire violation warning process after incremental update, realizing the implementation from classification result to violation warning. Its physical significance is that the final classification result includes the determined violation type and the corresponding confidence value. The preset violation threshold is a judgment standard set according to the severity of different violation types and the platform's detection requirements. By comparing the confidence value of the final classification result with the violation threshold, it is possible to accurately determine whether there are any violations in the live stream and the severity of the violations, thereby generating targeted warning messages, completing multiple types of violation warnings, and realizing the practical implementation of live stream content security monitoring. In terms of technical implementation, the preset violation thresholds are set by the live streaming platform according to the classification standards of violation types, adopting a categorized threshold strategy. For serious violations such as pornography and violence, the violation threshold is set to 0.7; for minor violations such as advertising and verbal slips, the violation threshold is set to 0.8; and for non-violation results, there is no threshold comparison requirement. This categorized threshold strategy can differentiate judgments based on the severity of the violation, avoiding over-warning or under-warning. First, the result type in the final classification result is matched with the violation threshold database. If the result type is non-violation, the live stream is determined to have no violation behavior and no warning is issued. If the result type is a certain violation type, the preset violation threshold corresponding to that type is extracted and set as the maximum threshold. The confidence value of the final classification result is compared with the threshold. If the confidence value exceeds the threshold, the live stream is determined to have this type of violation. The system's warning generation module will generate standardized warning information based on the determined violation type, confidence value, live stream identifier, timestamp, and other information. The warning information includes core content such as violation type, violation confidence, live stream room number, and warning time. Subsequently, the system's warning execution module will issue multiple types of violation warnings based on the content of the warning information. For minor violations, a pop-up reminder to the streamer is used. For serious violations, a pop-up reminder plus manual review and intervention is used. At the same time, the warning information is stored in the live stream violation ledger for easy follow-up violation tracing and platform management. For example, if the final classification result is a pornographic violation with a confidence value of 0.85, which exceeds the violation threshold of 0.7 for this type, the system will generate a warning message containing the violation type, confidence value, and live stream room number, and execute a warning operation involving pop-up reminder and manual review; if the final classification result is an advertising-related violation with a confidence value of 0.75, which does not exceed the violation threshold of 0.8 for this type, no warning message will be generated.The technical benefits of this step are that the categorized violation threshold strategy makes violation judgments more aligned with the platform's actual detection needs, enabling differentiated judgments for violations of varying severity. This avoids the problem of unreasonable warnings caused by a single threshold. The standardized generation of warning information and the hierarchical warning execution method make multi-type violation warnings more targeted and practical. It can promptly remind streamers to correct minor violations and quickly take manual review measures for serious violations, effectively improving the efficiency and effectiveness of live streaming content security monitoring. At the same time, the storage of violation ledgers provides data support for the platform's violation management and model optimization.

[0053] like Figure 2 As shown, this application also provides an incremental learning-based live streaming multi-type violation early warning system, including: The first acquisition module is used to acquire multimodal data of the live stream, wherein the multimodal data includes visual feature information, acoustic feature information and semantic feature information; The second acquisition module is used to acquire optical flow motion parameters and color histogram parameters based on the visual feature information, and to acquire spatiotemporal violation components based on the optical flow motion parameters and the color histogram parameters. The third acquisition module is used to acquire Mel frequency cepstral coefficients and fundamental frequency parameters based on the acoustic feature information, and to acquire abnormal audio components based on the Mel frequency cepstral coefficients and fundamental frequency parameters. The fourth acquisition module is used to acquire the text embedding vector as a semantic violation component based on the semantic feature information, and to perform weighted fusion based on the semantic violation component, the spatiotemporal violation component and the abnormal audio component to obtain a comprehensive confidence score; The judgment module is used to determine whether the overall confidence score is lower than a preset threshold; If the overall confidence score is lower than a preset threshold, the multimodal data is marked as a low-confidence sample. A corresponding multimodal feature extractor is obtained based on the low-confidence sample. The low-confidence sample is then subjected to small-sample learning to obtain a small-sample adaptation vector classification head. The multimodal feature extractor is then incrementally updated based on the adaptation vector classification head to obtain an incremental update result. Based on the incremental update result, multiple types of violation warnings are issued for the live stream.

[0054] In one embodiment, the second acquisition module includes: The first acquisition unit is used to acquire a continuous video frame sequence based on the visual feature information, acquire an optical flow field based on the continuous video frame sequence using the optical flow method, acquire the instantaneous motion speed and motion direction of the pixel based on the optical flow field, and statistically analyze the instantaneous motion speed and motion direction to obtain optical flow motion parameters. The second acquisition unit is used to acquire a single frame image based on the visual feature information, extract the distribution frequency of pixels in each channel of the RGB color space in the single frame image, construct a color histogram based on the distribution frequency, and calculate the peak value, mean and variance of the color histogram to obtain the color histogram parameters. The third acquisition unit is used to acquire a preset first spatiotemporal weight matrix, perform a dot product operation between the optical flow motion parameters and the first spatiotemporal weight matrix to obtain motion violation components, and perform a dot product operation between the color histogram parameters and the first spatiotemporal weight matrix to obtain color violation components. The first spatiotemporal weight matrix is ​​a preset weight vector that matches the dimensions of motion features and color features. The fourth acquisition unit is used to acquire a preset second spatiotemporal weight matrix and perform a weighted summation based on the motion violation component, the color violation component, and the second spatiotemporal weight matrix, wherein the second spatiotemporal weight matrix includes motion weights corresponding to the motion violation component and color weights corresponding to the color violation component, to obtain the spatiotemporal violation component.

[0055] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0056] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0057] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0058] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0059] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An incremental learning-based method for early warning of multiple types of violations in live streaming, characterized in that, include: Acquire multimodal data of a live stream, wherein the multimodal data includes visual feature information, acoustic feature information, and semantic feature information; Based on the visual feature information, optical flow motion parameters and color histogram parameters are obtained, and spatiotemporal violation components are obtained based on the optical flow motion parameters and color histogram parameters; Based on the acoustic feature information, obtain the Mel frequency cepstral coefficients and fundamental frequency parameters, and obtain the abnormal audio components based on the Mel frequency cepstral coefficients and fundamental frequency parameters; The text embedding vector is obtained as a semantic violation component based on the semantic feature information, and the semantic violation component, the spatiotemporal violation component and the abnormal audio component are weighted and fused to obtain a comprehensive confidence score. Determine whether the overall confidence score is lower than a preset threshold; If the overall confidence score is lower than a preset threshold, the multimodal data is marked as a low-confidence sample. A corresponding multimodal feature extractor is obtained based on the low-confidence sample. The low-confidence sample is then subjected to small-sample learning to obtain a small-sample adaptation vector classification head. The multimodal feature extractor is then incrementally updated based on the adaptation vector classification head to obtain an incremental update result. Based on the incremental update result, multiple types of violation warnings are issued for the live stream.

2. The incremental learning-based live streaming multi-type violation early warning method according to claim 1, characterized in that, The steps of obtaining optical flow motion parameters and color histogram parameters based on the visual feature information, and obtaining spatiotemporal violation components based on the optical flow motion parameters and color histogram parameters, include: A continuous video frame sequence is obtained based on the visual feature information, and an optical flow field is obtained based on the continuous video frame sequence using the optical flow method. The instantaneous motion velocity and motion direction of the pixels are obtained based on the optical flow field. The instantaneous motion velocity and motion direction are statistically analyzed to obtain the optical flow motion parameters. A single-frame image is obtained based on the visual feature information, and the distribution frequency of pixels in each channel of the RGB color space in the single-frame image is extracted. A color histogram is constructed based on the distribution frequency, and the peak value, mean, and variance of the color histogram are calculated to obtain the color histogram parameters. A preset first spatiotemporal weight matrix is ​​obtained. The optical flow motion parameters are multiplied by the first spatiotemporal weight matrix to obtain motion violation components. The color histogram parameters are multiplied by the first spatiotemporal weight matrix to obtain color violation components. The first spatiotemporal weight matrix is ​​a preset weight vector that matches the dimensions of motion features and color features. Obtain a preset second spatiotemporal weight matrix, and perform a weighted sum based on the motion violation component, the color violation component, and the second spatiotemporal weight matrix. The second spatiotemporal weight matrix includes motion weights corresponding to the motion violation component and color weights corresponding to the color violation component, thereby obtaining the spatiotemporal violation component.

3. The incremental learning-based live streaming multi-type violation early warning method according to claim 1, characterized in that, The steps of obtaining Mel frequency cepstral coefficients and fundamental frequency parameters based on the acoustic feature information, and obtaining anomalous audio components based on the Mel frequency cepstral coefficients and fundamental frequency parameters, include: The audio signal within a preset window is obtained based on the acoustic feature information. The audio signal is pre-emphasized, framed, and windowed to obtain a pre-processed audio frame. The pre-processed audio frame is subjected to a fast Fourier transform to obtain the spectrum. The spectrum is passed through a Mel filter bank, and the logarithm of the filtered energy is taken before a discrete cosine transform is performed to obtain the Mel frequency cepstral coefficients. Based on the audio signal, candidate fundamental frequency values ​​are obtained, and a continuous fundamental frequency trajectory is extracted from the candidate fundamental frequency values ​​using a dynamic programming algorithm. The fundamental frequency trajectory is then smoothed to obtain the fundamental frequency parameters. A preset audio violation detection model is obtained, wherein the audio violation detection model is a binary classification neural network model trained based on a large number of labeled violation and non-violation audio samples. The Mel frequency cepstral coefficients and the fundamental frequency parameters are input into the audio violation detection model to obtain the abnormal audio probability value output by the audio violation detection model. The abnormal audio probability value is used as the abnormal audio component.

4. The incremental learning-based live streaming multi-type violation early warning method according to claim 1, characterized in that, The step of obtaining a text embedding vector as a semantic violation component based on the semantic feature information, and then performing a weighted fusion of the semantic violation component, the spatiotemporal violation component, and the anomalous audio component to obtain a comprehensive confidence score includes: Based on the semantic feature information, obtain the real-time subtitle text or speech recognition text in the live stream, and perform word segmentation and stop word removal on the real-time subtitle text or speech recognition text to obtain preprocessed text; A pre-trained language representation model is obtained, the pre-processed text is input into the pre-trained language representation model, and the text embedding vector output by the pre-trained language representation model is obtained. The text embedding vector is used as a semantic violation component. Obtain a dynamic weight allocation strategy, wherein the dynamic weight allocation strategy is a mapping table that maps live streaming scene types to weight sets, and each weight set contains a semantic weight, a spatiotemporal weight and an audio weight. Identify the scene type of the current live stream, and obtain the semantic weight corresponding to the semantic violation component, the spatiotemporal weight corresponding to the spatiotemporal violation component, and the audio weight corresponding to the abnormal audio component from the dynamic weight allocation strategy according to the scene type; Based on the semantic weight, the spatiotemporal weight, and the audio weight, the semantic violation component, the spatiotemporal violation component, and the anomalous audio component are weighted and summed to obtain a comprehensive confidence score.

5. The incremental learning-based live streaming multi-type violation early warning method according to claim 1, characterized in that, The steps of performing small-sample learning on the low-confidence samples to obtain a small-sample adaptive vector classification head, and incrementally updating the multimodal feature extractor based on the adaptive vector classification head to obtain the incremental update result include: Obtain multimodal data of samples labeled as low-confidence samples, and use the multimodal data of the low-confidence samples as a support set; The support set is input into the multimodal feature extractor to obtain the feature vector of each sample in the support set; Obtain the violation type label for each sample in the support set, and group the feature vectors according to the violation type label, and calculate the prototype vector for each group of feature vectors; An adaptation vector classification head is constructed based on the prototype vector, wherein the adaptation vector classification head is used to calculate the distance between the feature vector of the sample to be classified and the prototype vector; A preset incremental update strategy is obtained. The adaptation vector classification head is integrated into the output of the multimodal feature extractor according to the incremental update strategy. Only the parameters of the last two layers of the multimodal feature extractor are fine-tuned, while the parameters of the remaining layers are fixed, to obtain the incremental update result.

6. The incremental learning-based live streaming multi-type violation early warning method according to claim 1, characterized in that, The step of issuing multi-type violation warnings for the live stream based on the incremental update results includes: Obtain the incremental update result, wherein the incremental update result includes the updated multimodal feature extractor and the newly added adaptation vector classification head; The updated multimodal feature extractor is used to obtain the feature vector of the new multimodal data in the live stream. Based on the multimodal data, obtain the corresponding preset original classification head, and input the feature vector into the preset original classification head and the newly added adaptation vector classification head respectively to obtain the original classification result and the adaptation classification result; A decision fusion rule is obtained, wherein the decision fusion rule is as follows: if the original classification result and the adapted classification result are consistent, then the original classification result is taken as the final classification result; if they are inconsistent, then the result with higher confidence is selected as the final classification result, and the original classification result and the adapted classification result are fused according to the decision fusion rule to obtain the final classification result; Based on the final classification result, it is determined whether there is any violation in the live stream. If the final classification result exceeds the preset violation threshold, an early warning message is generated, and multiple types of violation warnings are issued to the live stream based on the early warning message.

7. An incremental learning-based live streaming multi-type violation early warning system, characterized in that, include: The first acquisition module is used to acquire multimodal data of the live stream, wherein the multimodal data includes visual feature information, acoustic feature information and semantic feature information; The second acquisition module is used to acquire optical flow motion parameters and color histogram parameters based on the visual feature information, and to acquire spatiotemporal violation components based on the optical flow motion parameters and the color histogram parameters. The third acquisition module is used to acquire Mel frequency cepstral coefficients and fundamental frequency parameters based on the acoustic feature information, and to acquire abnormal audio components based on the Mel frequency cepstral coefficients and fundamental frequency parameters. The fourth acquisition module is used to acquire the text embedding vector as a semantic violation component based on the semantic feature information, and to perform weighted fusion based on the semantic violation component, the spatiotemporal violation component and the abnormal audio component to obtain a comprehensive confidence score; The judgment module is used to determine whether the overall confidence score is lower than a preset threshold; If the overall confidence score is lower than a preset threshold, the multimodal data is marked as a low-confidence sample. A corresponding multimodal feature extractor is obtained based on the low-confidence sample. The low-confidence sample is then subjected to small-sample learning to obtain a small-sample adaptation vector classification head. The multimodal feature extractor is then incrementally updated based on the adaptation vector classification head to obtain an incremental update result. Based on the incremental update result, multiple types of violation warnings are issued for the live stream.

8. The live streaming multi-type violation early warning system based on incremental learning according to claim 7, characterized in that, The second acquisition module includes: The first acquisition unit is used to acquire a continuous video frame sequence based on the visual feature information, acquire an optical flow field based on the continuous video frame sequence using the optical flow method, acquire the instantaneous motion speed and motion direction of the pixel based on the optical flow field, and statistically analyze the instantaneous motion speed and motion direction to obtain optical flow motion parameters. The second acquisition unit is used to acquire a single frame image based on the visual feature information, extract the distribution frequency of pixels in each channel of the RGB color space in the single frame image, construct a color histogram based on the distribution frequency, and calculate the peak value, mean and variance of the color histogram to obtain the color histogram parameters. The third acquisition unit is used to acquire a preset first spatiotemporal weight matrix, perform a dot product operation between the optical flow motion parameters and the first spatiotemporal weight matrix to obtain motion violation components, and perform a dot product operation between the color histogram parameters and the first spatiotemporal weight matrix to obtain color violation components. The first spatiotemporal weight matrix is ​​a preset weight vector that matches the dimensions of motion features and color features. The fourth acquisition unit is used to acquire a preset second spatiotemporal weight matrix and perform a weighted summation based on the motion violation component, the color violation component, and the second spatiotemporal weight matrix, wherein the second spatiotemporal weight matrix includes motion weights corresponding to the motion violation component and color weights corresponding to the color violation component, to obtain the spatiotemporal violation component.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images