Video violation identification method and device, electronic equipment and readable storage medium

Abstract

The invention provides a video violation identification method and apparatus, an electronic device and a readable storage medium. The method comprises the steps of obtaining a trained multi-modal large model; wherein the trained multi-modal large model is obtained by performing parameter fine adjustment on an original multi-modal large model based on violation image data; segmenting a plurality of image batches from the to-be-audited video; constructing prompt information for the image batch; wherein the prompt information comprises prior violation information of the image batch, a violation judgment rule of a specified violation category and output format constraint information of a violation recognition result; and carrying out violation identification on the image batch by the multi-modal large model according to the prior violation information and the violation judgment rule, and generating a violation identification result of the image batch according to the output format constraint information. According to the method, the effect of accurately identifying various complex video violation cases can be efficiently realized.

Description

Technical Field

[0001] This invention relates to the field of video review technology, and more specifically, to a method, apparatus, electronic device, and readable storage medium for identifying video violations. Background Technology

[0002] With the rapid development of live streaming and short video platforms, the massive amount of video uploads has made automated violation detection technology particularly important. Video data itself has highly complex characteristics, with significant variations in overall duration, ranging from a few seconds to tens of minutes or even longer. Within such videos, the time frame for violations is highly uncertain; some violation segments last only a few seconds, representing instantaneous risks, while others may persist throughout the entire video. Simultaneously, the range of violation categories that need to be identified is broad. This complex distribution in video length, violation duration, and violation category diversity makes it difficult for traditional automated review methods to balance recall and accuracy.

[0003] Currently, the mainstream solutions in the industry mainly fall into two categories. The first category is based on pure visual models. These methods are effective in identifying specific types of visual violations, but they struggle to understand complex violations related to context. The second category employs multimodal fusion models to fuse cross-modal information, thereby making an overall judgment on whether the entire video violates regulations. However, this approach faces challenges such as high deployment costs and low inference efficiency.

[0004] Therefore, there is an urgent need for an efficient video violation identification method to achieve comprehensive coverage and accurate identification of various complex violation scenarios. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a video violation identification method, device, electronic device and readable storage medium, which can efficiently and accurately identify a variety of complex video violation cases.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, the present invention provides a video violation identification method, the method comprising: obtaining a trained multimodal large model; wherein the multimodal large model is obtained by fine-tuning the parameters of an original multimodal large model based on violation images and image annotation data; segmenting multiple image batches from the video to be reviewed; constructing prompt information for the image batches; wherein the prompt information includes prior violation information of the image batches, violation judgment rules specifying violation categories, and output format constraint information of violation identification results; the multimodal large model performing violation identification on the image batches according to the prior violation information and the violation judgment rules, and generating violation identification results for the image batches according to the output format constraint information.

[0007] Secondly, the present invention provides a video violation identification device, comprising: an acquisition module for acquiring a trained multimodal large model; wherein the multimodal large model is obtained by fine-tuning the parameters of the original multimodal large model based on violation image data; a segmentation module for segmenting multiple image batches from the video to be reviewed; a construction module for constructing prompt information for the image batches; wherein the prompt information includes prior violation information of the image batches, violation judgment rules specifying violation categories, and output format constraint information of violation identification results; and an identification module for having the multimodal large model perform violation identification on the image batches according to the prior violation information and the violation judgment rules, and generate violation identification results for the image batches according to the output format constraint information.

[0008] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the video violation identification method described in any of the foregoing embodiments.

[0009] Fourthly, the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video violation identification method as described in any of the foregoing embodiments.

[0010] The video violation identification method, device, electronic device, and readable storage medium provided in this invention fine-tunes the parameters of an original multimodal large model based on a large amount of violation image data, thereby achieving a deep understanding of violation content in complex visual semantics and significantly improving the accuracy of identification. Furthermore, the video to be reviewed is segmented into multiple image batches according to a time sequence, and then prompt information is constructed for each image batch. This information integrates prior violation information obtained from context or historical behavior analysis, refined judgment rules for specified violation categories, and structured constraints on the output format, making the model's reasoning process more targeted and consistent. Finally, the multimodal large model integrates prior violation information and judgment rules to perform multi-dimensional violation identification on the image batches, and generates standardized violation identification results strictly according to the output format constraints. Overall, this achieves a higher accuracy and faster intelligent identification effect for various types of violation content in videos under complex scenarios.

[0011] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic flowchart illustrating the video violation identification method provided in this embodiment of the invention; Figure 2 This is a schematic diagram illustrating the generation of violation determination rules for a specified violation category provided in an embodiment of the present invention; Figure 3 This is an overall schematic diagram of the video violation identification process provided in an embodiment of the present invention; Figure 4 This is a functional block diagram of the video violation recognition device provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0015] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0016] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0017] First, let's introduce the relevant terminology used in the embodiments of this invention.

[0018] Video review: Video review is a content security detection process that uses automated algorithms and manual review to identify and process potentially illegal or risky content in videos, ensuring that videos published on the platform comply with laws and regulations, platform rules, and public order and good morals.

[0019] Multimodal large model: It is an artificial intelligence model that simultaneously receives, understands and generates multiple types of data (modalities, such as text, images, videos, audio, etc.). Based on the traditional large language model, it adds encoders for visual, speech and other modalities, so that the model can not only "understand text", but also "see images", "listen to audio" and "understand video", and fuse information from different modalities to perform cross-modal reasoning and generation.

[0020] Large language models: A type of natural language processing (NLP) model trained on massive amounts of text data and extremely large parameters. It can understand, generate, reason, and translate natural language, and even perform simple knowledge reasoning and creation.

[0021] Prompt engineering refers to the process of designing, optimizing, and managing input "prompts" used to drive large models to produce expected results.

[0022] Sensitive word matching: A rule-based review method that identifies all matching words in the content text according to a sensitive word database or a prohibited word database, and determines whether the content violates the rules.

[0023] On live streaming and short video platforms, tens of thousands of users upload video content every day. Some of this content is normal entertainment, chatting, or game streaming, but some may also contain inappropriate content. Platforms need to review these videos to identify and prevent such content from being disseminated.

[0024] Traditional methods employ a combination of machine and human review: content is first automatically screened using an identification model, and then suspicious results are reviewed by humans. However, the types of illegal content in the current online environment are numerous and unevenly distributed, with many categories of violations occurring very infrequently, exhibiting a significant "long-tail distribution" characteristic. Training a dedicated identification model for each type of illegal content would be costly and inefficient.

[0025] Therefore, existing technologies have shifted towards using multimodal large models for video violation identification, aiming to improve generalization ability and reduce system complexity by covering diverse violation categories with a unified model. However, multimodal large models still face the "information illusion" problem in practical applications, that is, there is an inconsistency between the actual content understood or extracted by the model and its final output judgment result, which may lead to misjudgment or omission, affecting the accuracy and reliability of the review.

[0026] Therefore, in order to improve the accuracy and efficiency of identifying various illegal content within videos, please refer to [link / reference needed]. Figure 1 , Figure 1 This is a schematic flowchart illustrating a video violation identification method provided in an embodiment of the present invention. The execution subject of this method can be an electronic device (such as a server), and includes steps S101 to S104: S101: Obtain the trained multimodal large model; wherein, the trained multimodal large model is obtained by fine-tuning the parameters of the original multimodal large model based on the violation image data; S102: Segment multiple image batches from the video to be reviewed; S103: Construct prompt information for image batches; wherein, the prompt information includes prior violation information of image batches, violation judgment rules specifying violation categories, and output format constraints of violation recognition results; S104: The multimodal large model performs violation identification on the image batch based on prior violation information and violation judgment rules, and generates the violation identification results of the image batch according to the output format constraint information.

[0027] Unlike existing technologies, this invention fine-tunes the parameters of the original multimodal large model based on a large amount of illegal image data, thereby achieving a deep understanding of illegal content in complex visual semantics and significantly improving the accuracy of recognition. Furthermore, the video to be reviewed is segmented into multiple image batches according to time sequence, and then prompts are constructed for each batch. These prompts integrate prior violation information obtained from context or historical behavior analysis, refined judgment rules for specified violation categories, and structured constraints on the output format, making the model's violation recognition process more targeted and consistent. Finally, the multimodal large model integrates prior violation information and judgment rules to perform multi-dimensional violation recognition on the image batches, and generates standardized violation recognition results strictly according to the output format constraints. Overall, this achieves higher accuracy and faster intelligent recognition of various types of illegal content in videos under complex scenarios.

[0028] Next, the embodiments of the present invention will be described in conjunction with the relevant accompanying drawings. Figure 1 The video demonstration provides a detailed explanation of the violation identification process.

[0029] In one embodiment of the present invention, to address the diverse types of violations and the presence of numerous rare categories (long-tail problem) and to improve the model's understanding of business scenarios, it is first necessary to obtain the original multimodal large model and labeled violation image data. Then, the original multimodal large model undergoes full parameter fine-tuning, thereby obtaining a multimodal large model optimized for business scenarios. Therefore, in step 101, this embodiment of the present invention can first select a multimodal large model from existing multimodal models. This model itself has the ability to understand image content and instructions. Furthermore, this embodiment of the present invention will, based on the unique review standards of the live streaming platform, enable the trained multimodal large model not only to identify what video frames are violations but also to generate violation recognition results according to a fixed format. The training method is as follows: Step a1: Obtain the illegal image and its prior illegal information; In this embodiment of the invention, the process begins by collecting image samples known to contain violations. These images are referred to as violation images, and each violation image is accompanied by violation category information pre-determined by either a human or a model. For example, a violation image may contain content that could be a violation of a certain type at the detection bounding box [707,194,787,357]. This information is called prior violation information, which tells the model exactly where the violation content appears in the image and what type of violation it belongs to.

[0030] In this embodiment of the invention, during the collection of violation images, the violation content within the platform can be analyzed and integrated to summarize various violation categories appearing on the platform, such as currency violations, text violations, long-tail violations, landmark violations, financial violations, suspected violations, and other violation categories. The violation categories can cover obvious violation categories, long-tail violation categories, and suspected violation categories. The goal of subsequent model fine-tuning is to enable the multimodal large model to understand business data and accurately identify various types of violation content.

[0031] In this embodiment of the invention, to guide the model to output violation identification results in a unified and standardized manner, standardized violation identification results can also be generated based on prior violation information according to output format constraint information. Here, output format constraint information refers to a pre-defined structured template that clarifies the key fields that must be included in the model output and their corresponding meanings. For example, the model needs to generate a dictionary-formatted result that conforms to a specific structure, which includes at least two fields: "Case Description" and "Judgment Result." The "Case Description" records the risk details observed by the model based on image content, including visual elements, textual information, and their contextual relationships; while the "Judgment Result" requires the model to select one or more of the most matching categories from a set of violation categories specified by the platform as the final judgment conclusion.

[0032] For example, for the prior violation information "Content that may be a violation was found inside the detection box [707,194,787,357]", the final violation identification result can be expressed as: {"Violation content description": "The image shows an adult male behaving indecently in a public place, with no obstruction or reasonable context explanation", "Judgment result": "Violation category - Public nudity"}.

[0033] Step a2: Based on prior violation information and preset recognition constraints and output format constraints, construct prompt data for violation images; In this embodiment of the invention, as described in the preceding steps, the prior violation information tells the model where the violation content appears in the image and what type of violation it belongs to. The identification constraints refer to a series of behavioral norms and technical requirements set to constrain multiple dimensions of the model's analysis process during the construction of the prompt data. These constraints guide the model to form a thought path that conforms to the business review logic. These conditions limit the identification methods, focus areas, and expression styles that the model should follow when analyzing images.

[0034] For example, the identification constraints could look like this: "Identification process requirements: 1. Extracted risk content / text must be retained (as a basis for analysis). 2. Combined with the overall analysis, your thought process must be reflected. 3. The language style should avoid politeness or avoiding sensitive words; it should be direct, accurate, and factual, around 100 words. 4. The judgment result should provide the subcategories within the violation categories (the names of the subcategories do not necessarily follow the above terms; identify the names based on the actual situation). If there are multiple violations, select the 1-2 most serious categories as representatives for the conclusion."

[0035] Optionally, in this embodiment of the invention, to guide the model to have a clear task positioning and professional perspective during the recognition process, the prompt data may also include the user role information of the large model for the user, enabling it to respond with the professional knowledge and judgment standards of the corresponding role when analyzing images. For example, "You are 'XXX,' an image review expert, skilled at combining images and text to discover abnormal intentions." With such a setting, the model will be more inclined to adopt the mindset of a reviewer when processing illegal images, focusing on high-risk details and making judgments in accordance with the platform's content security rules. In addition, auxiliary role information for illegal images can also be set to supplement the explanation of the identity attributes of other participants or content sources in the image, helping the model to better understand the image context.

[0036] The information above is used to construct the violation image alert data, which includes model role information, output format constraints, the location of the violation content in the image, the type of violation, and identification constraints. For example, the alert data might look like this: "You are 'XXX,' an image review expert, skilled at combining images and text to identify unusual intent and review this image." Does it comply with network communication security? The location detection results of Model A are as follows: Content that may be of a certain violation category was found at box [707,194,787,357]. Since model A only focuses on the image patch within the box and does not recognize the surrounding context, it has no certainty about the recognized category. Therefore, please describe in detail the content of the detected box (including image description and text near the box) to further review whether the image complies with network dissemination security.

[0037] The final required response format is: {"Description of Violation": "Description", "Judgment Result": "Violation Category"}. The category of the judgment result should be summarized in conjunction with the image. Please refer to the violation categories specified by the live streaming platform: currency, text violation, long-tail category, landmark, financial violation, suspected violation, etc.

[0038] Requirements for the identification process: 1. Extracted risk content / text must be retained (as a basis for analysis). 2. Combined with the overall analysis, your thought process must be demonstrated. 3. The language style should avoid politeness or avoiding sensitive words; direct, accurate, and factual text is required, approximately 100 words. 4. The judgment result should provide the sub-category of the violation (the names of the sub-categories do not necessarily follow the above terms; identify the names based on the actual situation). If there are multiple violations, select the 1-2 most serious categories as representative conclusions.

[0039] Step a3: Generate training data based on the violation image, cue data, and prior violation information; In this embodiment of the invention, each violation image can be used to construct a training data set, which is then used to fine-tune the parameters of the multimodal large model.

[0040] Step a4: Fine-tune the parameters of the multimodal large model using the training data to obtain the trained multimodal large model.

[0041] In this embodiment of the invention, the multimodal large model may include the following essential functional modules: a visual encoder, a cross-modal alignment layer, an instruction-driven language decoder, and a fine-tuning adapter. During training, after structured training samples (including image input, prompt data, and manually labeled violation categories) are input into the model, the visual encoder receives the violation images and maps them into a fixed-dimensional visual token sequence; the cross-modal alignment layer achieves spatial alignment and semantic fusion of visual tokens and prompt data through a cross-attention mechanism, explicitly modeling the relationship between local regions (such as content within the detection box) and global context (such as text surrounding the detection box); under the guidance of strongly constrained prompt data (such as "You are an image review expert" and "Output format must be JSON"), the language decoder generates a structured response, and the output case description and violation category are compared item by item with the manually labeled ground truth. Based on the differences between the two, a loss function is constructed, and all parameters of the model are updated by the parameter fine-tuning adapter using a full parameter fine-tuning strategy through backpropagation. These parameters include those of the visual encoder, cross-modal alignment layer, language decoder, and parameter fine-tuning adapter. The training process is continuously iterated, allowing the model to gradually learn to make accurate judgments based on image content until the preset training termination condition is met, resulting in the final multimodal large model used in this embodiment of the invention.

[0042] Based on the multimodal large model obtained through the above training method, this embodiment of the invention also designs a prompt word engineering, which can generate refined violation judgment rules for complex violation categories, enabling the model to stably identify various violation category cases. Specifically, as shown in steps b1 to b3: Step b1: Obtain multiple violation images and initial violation judgment rules corresponding to each specified violation category; In this embodiment of the invention, for each violation category specified by a live streaming platform, multiple (e.g., 100) typical violation image cases can be collected. Simultaneously, multiple existing violation descriptions for each specified violation category can be collected, and then these violation descriptions are summarized using a large model to obtain initial violation judgment rules. For example, for text violations, there are 70 violation descriptions, which are summarized to obtain 20 initial violation judgment rules.

[0043] Step b2: Use a multimodal large model to sequentially identify violations in each violation image and generate violation identification results; In this embodiment of the invention, the preceding training steps have already enabled the multimodal large model to make accurate violation judgments based on image content. Therefore, the multimodal large model is used here to identify each violation image of a specified violation category, generating a case description and violation category.

[0044] Step b3: Update the violation judgment rules based on each generated violation identification result until the final violation judgment rules are generated.

[0045] In this embodiment of the invention, based on the violation identification result of each violation image, it can be sequentially determined whether the current violation judgment rule needs to be updated. Specifically, this can be done according to the following steps: Step 1: Determine whether the violation category in the violation identification result corresponding to a certain violation image is correct; Step 2: If correct, keep the current violation judgment rule unchanged and continue to identify the next violation image; otherwise, generate a new violation judgment rule based on the violation content description in the violation identification result. Step 3: Update the current violation judgment rule with the new violation judgment rule or add the new violation judgment rule to the current violation judgment rule, and continue to identify the next violation image; Step 4: Use the updated violation determination rules as the final violation determination rules for the specified violation category.

[0046] This can be understood as follows: For each specified violation category, multiple violation images corresponding to it can be traversed, allowing the multimodal large model to identify violations and obtain case descriptions and violation categories. If the violation category identified by the model is correct, that is, the same as the specified violation category, then the current violation judgment rule remains unchanged. If the identified violation category is not the specified violation category, then the rule is summarized based on the violation content description output by the multimodal large model, and the existing violation judgment rules are updated or the rule summary is added to the current existing rules, until each violation image has been traversed.

[0047] For a better understanding of the above process, please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram illustrating the generation of violation judgment rules for specified violation categories provided in an embodiment of the present invention. After obtaining the violation judgment rules for each specified violation category in this way, the multimodal large model can correctly output the violation category according to these violation judgment rules during subsequent video review.

[0048] In one embodiment of the present invention, during the process of updating the violation judgment rules, the total number of violation judgment rules can be kept less than or equal to a preset number threshold. This can avoid the system performance degradation caused by the excessive expansion of the rule system and ensure the response efficiency and stability of the multimodal large model in high-concurrency scenarios.

[0049] Next, based on the multimodal large model obtained in the previous steps and the violation determination rules for the specified violation categories, embodiments of the present invention will be introduced. Figure 1 Steps S102 to S104 in the process.

[0050] In one embodiment of the present invention, the multimodal large model can be input as video, batch images, or single images. In step S102, in order to ensure that the multimodal large model can capture transient violations while controlling computational overhead, the present invention can first design a reasonable image input strategy, that is, to segment multiple image batches from the video to be reviewed, thereby obtaining structured video segment input units.

[0051] It should be understood that a video is composed of a series of images. If an entire video of several minutes is directly input into a multimodal large model, the model may experience excessive computational resource consumption and a significant increase in recognition latency due to the excessively long input sequence. Therefore, this embodiment of the invention requires slicing the video to be reviewed. To this end, the inventors discovered the following phenomena during their research: violations shorter than 1 second are extremely rare; when more than 3 images are input at once, the model sometimes forgets the image if even one of them is a violation; when 2-6 images are input at once, the processing speed is within an acceptable range. Based on these findings, this embodiment of the invention provides a video slicing processing method, as shown in steps c1 to c2: Step c1: Extract video frames from the video to be reviewed according to the preset time granularity; Step c2: Combine a preset number of consecutive video frames into an image batch.

[0052] In this embodiment of the invention, the preset time granularity can be 1 second, meaning one frame is extracted from the video every second. The preset number of frames can be set between 2 and 6, for example, 3 frames, which means combining 3 consecutive frames into a group as an image batch. This image batch is then sent to a multimodal large model for analysis. For example, a 10-second video will be cut into 3 complete image batches (each containing 3 frames), with the remaining frame processed separately. This processing method can accurately identify static violations, action violations, and instantaneous violations, ensuring that key information is not diluted and avoiding resource waste, thus achieving a balance between efficiency and effectiveness.

[0053] Furthermore, for each image batch in step S102, cue data is constructed in step S103. The "cue information" here is different from the cue words constructed when training the multimodal large model. In addition to including the prior violation information of the image batch and the output format constraint information of the violation recognition results, it also includes multiple violation judgment rules for each specified violation category in the aforementioned cue word project.

[0054] Under the prompting of step S103, when executing S104, the multimodal large model can identify violations in the image batch based on prior violation information and violation judgment rules, and then generate the violation identification results (including case description and violation category) of the image batch according to the output format constraint information, thereby achieving the purpose of correctly describing the violation case and correctly outputting the violation category specified by the platform.

[0055] In one embodiment of the present invention, to overcome the situation where the case description does not match the violation category (i.e., the multimodal large model exhibits a hallucination problem, where the case description correctly describes the violation points of the image, but the output violation category is incorrect), this embodiment of the present invention can also pre-construct a sensitive word library. Whenever a key word from the sensitive word library appears in the case description output by the multimodal large model, it is immediately marked as a violation. This method is fast-responding and updates quickly, making it suitable for dealing with breaking news events or new scams. Therefore, after obtaining the case descriptions for the image batch through steps S101 to S104, the video violation identification method provided by this embodiment of the present invention can also perform the following steps: Step d1: Check whether the keywords in the description of the illegal content are in the preset sensitive word database; Step d2: If yes, then correct the violation identification result based on the hit sensitive words; otherwise, retain the current violation identification result.

[0056] In this embodiment of the invention, the above-mentioned sensitive word matching process can be implemented using a large language model, and multiple results can be fused to make a final decision, thereby obtaining a comprehensive judgment result. The large language model used here can follow the same prompt word engineering as the multimodal large model, the difference being that the input of the large language model is a text description of the video content.

[0057] The above implementation methods can effectively alleviate the misjudgment problem caused by hallucination defects in multimodal large models.

[0058] For a better overall understanding of the above embodiments of the invention, please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the overall video violation identification process provided in an embodiment of the present invention. This video violation identification method can achieve the following technical effects: First, the embodiments of the present invention can significantly enhance the ability of live streaming platforms to identify video violations. By introducing the general understanding ability of large models, it can achieve efficient detection of various video scale-related violations.

[0059] Secondly, significant improvements were achieved in business metrics. Testing showed that the recall rate for general violations increased from 60% to 85%, and the recall rate for high-risk violations increased from 82% to over 95%. Compared to the traditional approach of continuously building a single identification model, this invention effectively reduces reliance on large-scale labeled data, significantly reduces manual labeling costs, and possesses the agile ability to quickly expand to include new violation categories, thus improving the system's maintainability and adaptability.

[0060] Finally, the application of the embodiments of the present invention can effectively promote the compliant operation of live streaming platforms, help better comply with community self-discipline guidelines, effectively maintain a clean cyberspace environment, enhance user trust and platform credibility, and promote the healthy and sustainable development of the live streaming industry.

[0061] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of the video violation detection device 40 is given below. Please refer to... Figure 4 , Figure 4 This is a functional block diagram of a video violation recognition device provided in an embodiment of the present invention. The video violation recognition device 40 includes: an acquisition module 401, a segmentation module 402, a construction module 403, and a recognition module 404.

[0062] The acquisition module 401 is used to obtain a trained multimodal large model; wherein, the multimodal large model is obtained by fine-tuning the parameters of the original multimodal large model based on the violation image data; The segmentation module 402 is used to segment multiple image batches from the video to be reviewed; The construction module 403 is used to construct prompt information for the image batch; wherein, the prompt information includes prior violation information of the image batch, violation judgment rules specifying the violation category, and output format constraint information of the violation recognition result; The identification module 404 is used to identify violations in the image batch by the multimodal large model according to the prior violation information and the violation judgment rules, and to generate the violation identification result of the image batch according to the output format constraint information.

[0063] It is understandable that the acquisition module 401, the segmentation module 402, the construction module 403, and the recognition module 404 can be executed collaboratively. Figure 1 Each step in the process is to achieve the corresponding technical effect.

[0064] It should be noted that the video violation identification device 40 provided in this embodiment of the invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this embodiment of the invention are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiments can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0065] Optionally, the above modules can be stored in the form of software or firmware. Figure 5 The memory shown is either stored in or embedded in the operating system (OS) of the electronic device 50, and can be used by... Figure 5 The processor executes the commands. Meanwhile, the data and program code required to execute these modules can be stored in memory.

[0066] Please see Figure 5 , Figure 5 The diagram illustrates a structural block diagram of an electronic device provided in an embodiment of the present invention, including a memory 501, a processor 502, and a communication interface 503. The memory 501, processor 502, and communication interface 503 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0067] Optionally, the bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0068] In this embodiment of the invention, the processor 502 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment of the invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The software modules may be located in the memory 501, and the processor 502 reads the program instructions from the memory 501 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0069] In this embodiment of the invention, the memory 501 can be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or it can be volatile memory, such as RAM. The memory can also be any other medium capable of carrying or storing desired executable program code having an instruction or data structure form and accessible by a computer, but is not limited thereto. The memory in this embodiment of the invention can also be a circuit or any other device capable of implementing a storage function for storing instructions and / or data.

[0070] The memory 501 can be used to store software programs and modules, such as the instructions / modules of the video violation identification device 40 provided in this embodiment of the invention. These can be stored in the memory 501 in the form of software or firmware, or embedded in the operating system (OS) of the electronic device 50. The processor 502 executes various functional applications and data processing by executing the software programs and modules stored in the memory 501. The communication interface 503 can be used to communicate with other node devices for signaling or data.

[0071] Understandable. Figure 5 The structure shown is for illustrative purposes only; the electronic device 50 may also include components that are more advanced than those shown. Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown. Figure 5 The components shown can be implemented using hardware, software, or a combination thereof.

[0072] Based on the above embodiments, the present invention also provides a storage medium in which a computer program is stored. When the computer program is executed by a computer, the computer executes the video violation identification method provided in the above embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0073] Based on the above embodiments, the present invention also provides a program product, which includes a computer program. The processor can execute the computer program to implement the video violation identification method provided in the embodiments of the present invention. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0074] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0075] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs.

[0076] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0077] It should be noted that if the function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying video violations, characterized in that, The method includes: A trained multimodal large model is obtained; wherein the trained multimodal large model is obtained by fine-tuning the parameters of the original multimodal large model based on the violation image data; Segment the video to be reviewed into multiple image batches; Construct prompt information for the image batch; wherein, the prompt information includes prior violation information of the image batch, violation judgment rules specifying the violation category, and output format constraint information of the violation recognition result; The multimodal large model performs violation identification on the image batch based on the prior violation information and the violation judgment rules, and generates the violation identification result of the image batch according to the output format constraint information.

2. The video violation identification method according to claim 1, characterized in that, Obtain a trained multimodal large model, including: Obtain the illegal image and the prior illegal information of the illegal image; Based on the prior violation information, preset recognition constraints, and output format constraints, prompt data for the violation image is constructed. Training data is generated based on the violation image, the prompt data, and the prior violation information; The multimodal large model is fine-tuned using the training data to obtain the trained multimodal large model.

3. The video violation identification method according to claim 1, characterized in that, The method further includes: Obtain multiple violation images and initial violation judgment rules corresponding to each specified violation category; The multimodal large model is used to sequentially identify violations in each violation image, generating violation identification results; The violation determination rule is updated based on each generated violation identification result until the final violation determination rule is generated.

4. The video violation identification method according to claim 3, characterized in that, The violation determination rule is updated based on each generated violation identification result until the final violation determination rule is generated, including: Determine whether the violation category in the violation identification result corresponding to a certain violation image is correct; If correct, maintain the current violation judgment rule and continue to identify the next violating image; otherwise, generate a new violation judgment rule based on the description of the violation content in the violation identification result. Update the current violation judgment rule with the new violation judgment rule or add the new violation judgment rule to the current violation judgment rule, and continue to identify the next violation image; The updated violation determination rules will be used as the final violation determination rules for the specified violation category.

5. The video violation identification method according to claim 3, characterized in that, The method further includes: During the process of updating the violation judgment rules, the total number of violation judgment rules should be kept less than or equal to the preset number threshold.

6. The video violation identification method according to claim 1, characterized in that, Multiple image batches are extracted from the video to be reviewed, including: Extract video frames from the video to be reviewed according to the preset time granularity; A preset number of consecutive video frames are combined into an image batch.

7. The video violation identification method according to any one of claims 1-6, characterized in that, The violation identification result includes a description of the violation content and a violation category; the method further includes: Check whether the keywords in the description of the illegal content are in a preset sensitive word database; If so, the violation identification result is corrected based on the detected sensitive words; otherwise, the current violation identification result is retained.

8. A video violation detection device, characterized in that, include: The acquisition module is used to obtain a trained multimodal large model; wherein the multimodal large model is obtained by fine-tuning the parameters of the original multimodal large model based on the violation image data; The segmentation module is used to segment multiple image batches from the video to be reviewed; A construction module is used to construct prompt information for the image batch; wherein, the prompt information includes prior violation information of the image batch, violation judgment rules specifying the violation category, and output format constraint information of the violation recognition result; The identification module is used to identify violations in the image batch by the multimodal large model based on the prior violation information and the violation judgment rules, and to generate the violation identification result of the image batch according to the output format constraint information.

9. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor to implement the video violation identification method according to any one of claims 1-7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video violation identification method as described in any one of claims 1-7.

Images