A video auditing method, device, equipment and medium

Abstract

A video auditing method, device, equipment and medium are disclosed. The method comprises: obtaining a video to be audited; performing frame extraction processing on the video to be audited according to the video duration of the video to be audited to obtain a target frame sequence; the target frame sequence comprises at least one target frame; performing text extraction on the video to be audited to obtain target text; inputting the target frame sequence and the target text into an initial multi-modal large model; if there is an abnormality in the output result of the initial multi-modal large model, adjusting the parameters of the initial multi-modal large model to obtain a target multi-modal large model, keeping the target frame sequence unchanged, inputting the target frame sequence and the target text into the target multi-modal large model, and obtaining a target auditing result. The embodiments of the present application can improve the stability of video auditing.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video review method, apparatus, device, and medium. Background Technology

[0002] As short videos become a crucial medium for brand promotion, content marketing, and crowdsourced content production, an increasing number of platforms need to conduct content compliance reviews of user-uploaded videos to determine whether they meet review requirements such as brand display guidelines, dress codes, safety behavior requirements, privacy protection requirements, and content dissemination guidelines. Since video content typically contains multiple modalities of data, including images, audio, subtitles, and user information, manual review alone is insufficient to meet the review needs of massive amounts of video content. Therefore, utilizing multimodal artificial intelligence technology to achieve automated video content review has become an important development direction in this field.

[0003] Currently, existing technologies typically use fixed, large models to review videos.

[0004] However, existing review methods suffer from low stability in video review, which can easily lead to interruptions in the review process. Summary of the Invention

[0005] This invention provides a video review method, apparatus, device, and medium. The embodiments of this invention can improve the stability of video review.

[0006] In a first aspect, embodiments of the present invention provide a video review method, the method comprising:

[0007] Obtain the videos pending review and the user-related information for those videos;

[0008] Based on the video length of the video to be reviewed, the video to be reviewed is subjected to frame extraction to obtain a target frame sequence; the target frame sequence includes at least one target frame.

[0009] Extract text from the video to be reviewed to obtain the target text;

[0010] Input the target frame sequence, target text, and user-related information into the initial multimodal large model;

[0011] If the initial multimodal large model has anomalies in the output process, the parameters of the initial multimodal large model are adjusted to obtain the target multimodal large model. The target frame sequence, target text and user association information are then input into the target multimodal large model to obtain the target review result.

[0012] Secondly, embodiments of the present invention also provide a video review device, the device comprising:

[0013] The data acquisition module is used to acquire the videos to be reviewed and the user-related information of the videos to be reviewed;

[0014] The video frame extraction module is used to extract frames from the video to be reviewed based on the video duration, and obtain a target frame sequence; the target frame sequence includes at least one target frame.

[0015] The target text extraction module is used to extract text from the video to be reviewed, and obtain the target text.

[0016] The data input module is used to input the target frame sequence, target text, and user-related information into the initial multimodal large model;

[0017] The target review result generation module is used to adjust the parameters of the initial multimodal large model if there are anomalies in the output results, to obtain the target multimodal large model, and input the target frame sequence, target text and user association information into the target multimodal large model to obtain the target review result.

[0018] Thirdly, embodiments of the present invention also provide a video review device, the video review device comprising:

[0019] At least one processor; and

[0020] A memory that is communicatively connected to at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the video review method of any embodiment of the present invention.

[0022] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions that are used to cause a processor to execute the video review method of any embodiment of the present invention.

[0023] Fifthly, embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements a video review method according to any embodiment of the present invention.

[0024] The technical solution of this invention acquires the video to be reviewed, user-related information, and video text information, and combines this with a target frame sequence obtained based on the video duration. The video image information and text information are then fused and input into a multimodal large model for review. This allows for comprehensive analysis of the video to be reviewed using multimodal information, thereby obtaining the corresponding video review result. When anomalies occur in the output results of the initial multimodal large model, the parameters of the initial multimodal large model are adjusted, and the review is re-executed based on the adjusted target multimodal large model. This helps avoid the direct termination of the review task due to abnormal model output, allowing the video review process to continue, thus improving the stability and continuity of the video review process. This solves the technical problem of low stability in existing video review technologies, thereby improving the stability of video review.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a video review method provided in an embodiment of the present invention;

[0028] Figure 2 A detailed flowchart of a fault-tolerant call for a video review method provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of a video review device provided in an embodiment of the present invention;

[0030] Figure 4 This is a structural schematic diagram of a video review device provided in an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] In the technical solutions of this invention, the acquisition, storage, and application of the videos to be reviewed and their associated user information comply with relevant laws and regulations and do not violate public order and good morals.

[0034] Figure 1 This is a general flowchart of a video review method provided by an embodiment of the present invention. This embodiment of the present invention is applicable to situations where platforms need to review the content compliance of massive amounts of videos. The method can be executed by a video review device, which can be implemented in hardware and / or software.

[0035] See Figure 1 The video review methods shown include:

[0036] S101. Obtain the video to be reviewed.

[0037] Optionally, in addition to obtaining the videos to be reviewed, you can also obtain the user association information of the videos to be reviewed.

[0038] User association information refers to information related to the uploading user or publishing entity corresponding to the video to be reviewed, reflecting the relationship between the video and the corresponding user. User association information may include user identification, account attributes, business identity, organization affiliation, role type, and other information that describes the video's source. User association information is typically stored and transmitted in the form of fields, key-value pairs, or structured data, and can establish a unique association with the corresponding video to be reviewed.

[0039] S102. Based on the video duration of the video to be reviewed, perform frame extraction processing on the video to be reviewed to obtain a target frame sequence; the target frame sequence includes at least one target frame.

[0040] Frame extraction, in particular, refers to the process of extracting a portion of video footage from a video to be reviewed. Its purpose is to represent the entire video content using representative video frames, without needing to analyze all the frames in the video. Frame extraction determines the appropriate extraction interval based on the video's duration and extracts video frames accordingly, ensuring that videos of different lengths can obtain video frame data appropriate to their duration. Frame extraction can employ a fast-seek method based on a target timestamp to approximate the location of keyframes near the target time point. This avoids the need for complete decoding of large amounts of irrelevant video data, as is done in traditional methods, thus improving the efficiency of video frame extraction. The target time point is a preset time point.

[0041] The target frame sequence refers to an image sequence composed of multiple target frames obtained after frame extraction, arranged in chronological order of video playback. The target frame sequence includes at least one target frame, each corresponding to a specific time point in the video to be reviewed, and retaining the content of the video at that corresponding time point. As a crucial visual input to the multimodal large model, the target frame sequence enables the model to comprehensively understand the entire video content based on video images from multiple time points, rather than relying solely on a single image, thereby improving the completeness of visual information during video review. For example, if a video to be reviewed yields twenty target frames after frame extraction, these twenty target frames are arranged in the video playback order to form the target frame sequence.

[0042] Specifically, different frame skipping intervals can be set according to different video lengths. For example, when the video length is greater than or equal to 10 seconds, a 2-second frame skipping interval can be used, and when the video length is less than 10 seconds, a 1-second frame skipping interval can be used.

[0043] S103. Extract text from the video to be reviewed to obtain the target text.

[0044] Text extraction refers to the process of obtaining text information that reflects the semantic content of a video from a video to be reviewed. The main objects of extraction include the text information corresponding to the audio content in the video. Text extraction is used to convert the audio information in the video into text form, and together with the video frame information, it serves as multimodal input, improving the ability of the multimodal large model to understand the video content.

[0045] The target text refers to text data that has undergone text extraction and cleaning processes and can be used for video content review. It mainly reflects the semantic information in the video to be reviewed. Compared with the initial text, the target text has removed non-text characters and abnormal content such as injected text, and has a higher degree of standardization and readability.

[0046] S104. Input the target frame sequence and target text into the initial multimodal large model.

[0047] Optionally, when obtaining user association information for the video to be reviewed, the target frame sequence, target text, and user association information can be input into the initial multimodal large model.

[0048] The initial multimodal big model can refer to a multimodal artificial intelligence model used for the first execution of video review tasks. The initial multimodal big model can receive data from different modalities and perform joint understanding and reasoning on information from different modalities.

[0049] S105. If there are anomalies in the output results of the initial multimodal large model, the parameters of the initial multimodal large model are adjusted to obtain the target multimodal large model. The target frame sequence is kept unchanged. The target frame sequence and target text are input into the target multimodal large model to obtain the target review result.

[0050] The target multimodal large model can refer to the multimodal large model obtained by adjusting the model calling parameters after the initial multimodal large model has an output anomaly, which is used to re-execute the video review task.

[0051] The target review result refers to the final video review result determined after video content analysis by a multimodal large model, used to characterize the review conclusion corresponding to the video to be reviewed. In this invention, the target review result serves as the final output of the entire video review process, providing a basis for subsequent business processes such as video review triage, automatic approval, automatic rejection, or manual review.

[0052] Specifically, when the initial multimodal large model fails to output results or the output of the initial multimodal large model is abnormal, the parameters of the initial multimodal large model can be adjusted to obtain the target multimodal large model.

[0053] In one optional embodiment, existing multimodal large model review schemes, in pursuit of stability and reproducibility of output results, typically set the model's sampling temperature parameter to zero, making the model output deterministic—that is, under the condition that the input remains unchanged, the output result of the model is consistent in each call. However, this deterministic characteristic can lead to a deterministic failure deadlock problem in call failure scenarios: when a call produces a formatted error response, empty response, parsing failure, or timeout due to the model's internal inference entering a specific path, due to the determinism of the model output, retrying the call with the same parameters will re-execute along the same deterministic path, producing the exact same failure result. Retry cannot heal itself, ultimately causing the entire video review task to fail, requiring manual processing or direct discarding, resulting in wasted computing power and increased review latency. To address the above shortcomings, this invention proposes: while keeping the input frame sequence unchanged (i.e., without reducing the integrity of the review input information), apply a small perturbation to the sampling temperature parameter of the multimodal large model, so that the model completes inference along a different sampling path in the next call, thereby escaping the deterministic failure path of the previous call and achieving self-healing fault tolerance.

[0054] As can be seen, in this embodiment, by acquiring the video to be reviewed, user-related information, and video text information, and combining this with the target frame sequence obtained based on the video duration, the video image information, text information, and user-related information are fused and input into a multimodal large model for review. This allows for comprehensive analysis of the video to be reviewed using multimodal information, thereby obtaining the corresponding video review result. When anomalies occur in the output results of the initial multimodal large model, the parameters of the initial multimodal large model are adjusted, and the review is re-executed based on the adjusted target multimodal large model. This helps avoid the direct termination of the review task due to abnormal model output, allowing the video review process to continue, thereby improving the stability and continuity of the video review process. This solves the technical problem of low stability in existing video review technologies, thus improving the stability of video review.

[0055] In an optional embodiment, Figure 2The following is a detailed flowchart of the fault-tolerant call process for a video review method provided in this embodiment of the invention. The original statement, "If an anomaly occurs in the output of the initial multimodal large model, adjust the parameters of the initial multimodal large model to obtain the target multimodal large model, while keeping the target frame sequence unchanged, input the target frame sequence and target text into the target multimodal large model to obtain the target review result," is further refined into: "Adjust the model call parameters corresponding to the initial multimodal large model to obtain the first candidate large model, input the target frame sequence and target text into the first candidate large model to obtain the first candidate review result; if the first candidate review result meets the review rules, then the first candidate review result is used as the target review result." If the first candidate review result does not meet the review rules, the model calling parameters corresponding to the first candidate large model are adjusted to obtain the second candidate large model. The target frame sequence and target text are then input into the second candidate large model to obtain the second candidate review result. If the second candidate review result meets the review rules, it is used as the target review result. If the second candidate review result does not meet the review rules, the model calling parameters corresponding to the second candidate large model are adjusted to obtain the third candidate large model. The target frame sequence and target text are then input into the third candidate large model to obtain the third candidate review result, which is then used as the target review result. This process is designed to improve the operation of the video review method.

[0056] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the descriptions in other embodiments.

[0057] See Figure 2 The video review methods shown include:

[0058] S201. Obtain the video to be reviewed.

[0059] S202. Based on the video duration of the video to be reviewed, perform frame extraction processing on the video to be reviewed to obtain a target frame sequence; the target frame sequence includes at least one target frame.

[0060] S203. Extract text from the video to be reviewed to obtain the target text.

[0061] S204. Input the target frame sequence and target text into the initial multimodal large model.

[0062] S205. Adjust the model calling parameters corresponding to the initial multimodal large model to obtain the first candidate large model, and input the target frame sequence and target text into the first candidate large model to obtain the first candidate review result.

[0063] The first candidate large model can refer to the first-stage review model obtained by adjusting the model call parameters corresponding to the initial multimodal large model after the initial multimodal large model has an output anomaly. The first candidate large model is used to re-execute the video review task.

[0064] The first candidate review result can refer to the review result output by the first candidate big model after completing the video review. The first candidate review result is used to characterize the review conclusion of the first candidate big model on the video to be reviewed.

[0065] S206. If the first candidate audit result meets the audit rules, then the first candidate audit result shall be used as the target audit result.

[0066] Among them, the review rules can refer to the judgment rules used to determine whether the output results of a multimodal large model meet the preset requirements.

[0067] S207. If the first candidate review result does not meet the review rules, adjust the model call parameters corresponding to the first candidate large model to obtain the second candidate large model, and input the target frame sequence and target text into the second candidate large model to obtain the second candidate review result.

[0068] The second candidate model can refer to the second-stage review model obtained by adjusting the model call parameters corresponding to the first candidate model when the first candidate review result does not meet the review rules. The second candidate model is used to continue to perform the video review task.

[0069] The second candidate review result can refer to the review result output by the second candidate big model after completing the video review, which reflects the video review conclusion obtained by the second candidate big model based on the parameters called by the current model.

[0070] S208. If the second candidate audit result meets the audit rules, then the second candidate audit result shall be used as the target audit result.

[0071] S209. If the second candidate review result does not meet the review rules, adjust the model call parameters corresponding to the second candidate large model to obtain the third candidate large model, and input the target frame sequence and target text into the third candidate large model to obtain the third candidate review result, and use the third candidate review result as the target review result.

[0072] The third candidate large model can refer to a multimodal large model obtained by readjusting the model call parameters corresponding to the second candidate large model when the second candidate review result still does not meet the review rules. The third candidate large model is used to continue the review task of the videos to be reviewed.

[0073] The third candidate review result can refer to the review result output by the third candidate big model after completing the video review based on the target frame sequence, target text and user-related information.

[0074] As can be seen, in this embodiment, when anomalies occur during the output of the initial multimodal large model, the model calling parameters are adjusted sequentially to generate the first, second, and third candidate large models. Each candidate review result is then validated against the review rules. If a candidate review result meets the review rules, it is determined as the target review result; otherwise, the next candidate large model is used for review. This allows for continued video review using different model calling parameters even when model output is abnormal or the review result does not meet the review requirements. This helps reduce the likelihood of the review process being terminated due to a single abnormal model call, improving the continuity and stability of the video review process and giving the model calling process a certain degree of fault tolerance, thereby increasing the completion rate of the video review task to some extent.

[0075] In some embodiments, the model call parameters include: temperature parameters, call waiting time, and start / stop status of structured output constraints;

[0076] Adjusting the model invocation parameters corresponding to the initial multimodal large model yields the first candidate large model, including:

[0077] Set the temperature parameters of the initial multimodal large model to zero, set the call wait time to zero, and set the start / stop status of the structured output constraints to on to obtain the first candidate large model;

[0078] Adjusting the model invocation parameters corresponding to the first candidate large model yields the second candidate large model, including:

[0079] Set the temperature parameter of the first candidate large model to the first temperature value, and set the call wait time to the first duration to obtain the second candidate large model; the first temperature value is greater than zero.

[0080] Adjusting the model invocation parameters corresponding to the second candidate large model yields the third candidate large model, including:

[0081] The temperature parameter of the second candidate large model is set to zero, the call waiting time is set to the second duration, and the start / stop status of the structured output constraint is set to off, resulting in the third candidate large model; the duration of the second duration is longer than the duration of the first duration.

[0082] Optionally, temperature perturbation can be used to allow the model to escape the deterministic failure path of the previous call.

[0083] Temperature parameter settings refer to the process of configuring or adjusting temperature parameters during multimodal large model inference, which controls the degree of randomness in the model's generated results. When the temperature parameter is small, the model output usually has good determinism; when the temperature parameter is appropriately increased, the model output has a certain degree of randomness.

[0084] The call waiting time can refer to the waiting time parameter set before or during the process of calling a multimodal large model to perform an audit task. The call waiting time is used to control the time interval between model call requests.

[0085] The enabled / disabled status of structured output constraints indicates whether the multimodal large model enables structured output format constraints when generating review results. This controls whether the model's output meets preset data structure requirements. The enabled / disabled status typically includes an "on" state and a "off" state. An "on" state indicates that structured output constraints are enabled when calling the multimodal large model, causing the model to output review results according to a preset structure. A "off" state indicates that structured output constraints are not enabled, causing the model to output review results using plain text. When the initial multimodal large model experiences output anomalies, this status can be adjusted to change the model's calling method, allowing the model to regenerate review results using different output strategies, thereby improving the model's fault tolerance in abnormal situations. For example, if the initial multimodal large model returns an abnormal result after enabling structured output constraints, the structured output constraints can be disabled during subsequent candidate large model calls, causing the model to output review results in an unstructured manner; or the structured output constraints can be re-enabled when the model interface support conditions are met to obtain output results that meet the review rules, resolving the issue of endpoint incompatibility. For example, structured constraints may include JSON mode.

[0086] As can be seen, in this embodiment, by setting the model call parameters to temperature parameters and call waiting time, and using different parameter combinations for different candidate large models, the multimodal large model performs video review in different call methods at different call stages. This allows the model to use differentiated model call parameters for review at different call stages, which helps to avoid using the same call parameters for review at each stage and provides conditions for the model to complete inference again under different call conditions.

[0087] In some embodiments, based on the video duration of the video to be reviewed, frame extraction processing is performed on the video to be reviewed to obtain a target frame sequence, including:

[0088] The frame extraction interval of the video to be reviewed is determined based on the video's duration.

[0089] Based on the frame extraction interval, the video to be reviewed is processed by extracting frames to obtain the target frame sequence.

[0090] The frame extraction interval refers to the time interval between two consecutive target frames extracted during the frame extraction process of the video to be reviewed. The frame extraction interval is determined based on the video length of the video to be reviewed, allowing different frame extraction strategies to be used for videos of different lengths. This results in a target frame sequence that matches the video length, providing representative video image information for subsequent multimodal large-scale modeling.

[0091] As can be seen, in this embodiment, by determining the corresponding frame extraction interval based on the video duration of the video to be reviewed, and performing frame extraction processing on the video to be reviewed based on the determined frame extraction interval, the target frame sequence can be obtained by using a frame extraction method that is adapted to the duration of the video to be reviewed, thus providing a more reasonable visual input for subsequent multimodal large model video content analysis.

[0092] In some embodiments, text extraction is performed on the video to be reviewed to obtain target text, including:

[0093] Extract the audio data corresponding to the video to be reviewed;

[0094] The audio data is processed by speech recognition to obtain the initial text;

[0095] Perform character detection on the initial text to identify at least one non-text character in the initial text;

[0096] Delete the non-text characters to obtain the candidate text;

[0097] Based on a preset regular expression, pattern matching is performed on the candidate text to determine at least one injection text among the candidate texts;

[0098] Delete the injected text to obtain the target text.

[0099] Audio data can refer to the sound information contained in the video to be reviewed, which is used to carry the voice, background sound or other sound content in the video.

[0100] The initial text refers to the text content directly obtained after speech recognition processing of the audio data corresponding to the video to be reviewed. It reflects the basic semantic information of the speech expression in the video. The initial text is usually composed of letters, numbers, punctuation marks, and other recognized characters. The initial text provides raw text data for subsequent character detection and text cleaning, enabling the system to further delete non-text characters and injected text to obtain a more standardized target text. For example, for the audio content in a video, after speech recognition, the result is "Welcome @@ to ### this event!!!", which can be used as the initial text and further processed for character detection and text cleaning.

[0101] Non-text characters refer to characters that exist in the initial text but do not belong to the normal text content. Non-text characters usually cannot express effective semantic information or may affect subsequent text processing and model understanding.

[0102] The preset regular expressions refer to pre-configured character matching rules used to match and identify candidate text according to predetermined character patterns, thereby determining the text content in the candidate text that conforms to specific characteristics. The preset regular expressions are mainly used for pattern matching of candidate text to identify injected text and provide a basis for subsequent deletion of injected text, ensuring that the target text maintains good standardization and authenticity.

[0103] Injected text refers to text content that does not belong to the true semantic content of the video or may affect subsequent model review. Injected text is usually abnormal text that needs to be removed from the candidate text. As an object to be removed during the text cleaning process, injected text is removed from the candidate text after it is identified, thereby obtaining the target text. This reduces the impact of abnormal text on the semantic understanding of the multimodal large model and improves the standardization of the input text in the subsequent video review process.

[0104] As can be seen, in this embodiment, by extracting the audio data corresponding to the video to be reviewed and performing speech recognition processing, the initial text reflecting the semantic content of the video is obtained. Furthermore, non-text characters in the initial text and injected text in the candidate text are detected and deleted to obtain the target text. This can standardize the text obtained from speech recognition, which helps to reduce the impact of abnormal characters and abnormal text on the subsequent multimodal large model processing process and improve the standardization and effectiveness of the input text.

[0105] In some embodiments, before obtaining the video to be reviewed, the method further includes:

[0106] Get the video duration of the initial video;

[0107] When the duration of the initial video is greater than or equal to the duration threshold, the initial video is split into at least two initial sub-videos, and the initial sub-videos are identified as videos to be reviewed.

[0108] Specifically, when the duration threshold is 10 seconds, if the duration of the initial video is 12 seconds, then the duration of the initial video exceeds the duration threshold. The initial video is then split into two initial sub-videos with a duration of 6 seconds each, and these two initial sub-videos are designated as videos to be reviewed.

[0109] As can be seen, in this embodiment, by obtaining the video duration of the initial video before video review, and dividing the initial video into at least two initial sub-videos as videos to be reviewed when the video duration of the initial video is greater than or equal to the duration threshold, the long video can be divided into multiple relatively independent video segments for subsequent review. This helps to avoid treating the entire long video as a single review object and improves the flexibility of subsequent video review processing.

[0110] Figure 3 This invention provides a schematic diagram of a video review device. This invention is applicable to situations where platforms need to review the content compliance of massive amounts of videos. The device can execute video review methods and can be implemented in hardware and / or software.

[0111] See Figure 3 The video review device shown includes: a data acquisition module 301, a video frame extraction module 302, a target text extraction module 303, a data input module 304, and a target review result generation module 305, wherein...

[0112] Data acquisition module 301 is used to acquire videos to be reviewed;

[0113] The video frame extraction module 302 is used to extract frames from the video to be reviewed based on the video duration of the video to be reviewed, and obtain a target frame sequence; the target frame sequence includes at least one target frame.

[0114] The target text extraction module 303 is used to extract text from the video to be reviewed and obtain the target text.

[0115] Data input module 304 is used to input the target frame sequence and target text into the initial multimodal large model;

[0116] The target review result generation module 305 is used to adjust the parameters of the initial multimodal large model if there is an anomaly in the output result process, to obtain the target multimodal large model, while keeping the target frame sequence unchanged, and inputting the target frame sequence and target text into the target multimodal large model to obtain the target review result.

[0117] The technical solution of this invention acquires the video to be reviewed, user-related information, and video text information, and combines this with a target frame sequence obtained based on the video duration. The video image information, text information, and user-related information are then fused and input into a multimodal large model for review. This allows for comprehensive analysis of the video to be reviewed using multimodal information, thereby obtaining the corresponding video review result. When anomalies occur in the output results of the initial multimodal large model, the parameters of the initial multimodal large model are adjusted, and the review is re-executed based on the adjusted target multimodal large model. This helps avoid the direct termination of the review task due to abnormal model output, allowing the video review process to continue, thus improving the stability and continuity of the video review process. This solves the technical problem of low stability in existing video review technologies, thereby improving the stability of video review.

[0118] In some embodiments, if an anomaly occurs in the output of the initial multimodal large model, the parameters of the initial multimodal large model are adjusted to obtain the target multimodal large model, while keeping the target frame sequence unchanged. The target frame sequence and target text are input into the target multimodal large model to obtain the target review result. Specifically, the target review result generation module 305 is used for:

[0119] Adjust the model calling parameters corresponding to the initial multimodal large model to obtain the first candidate large model, and input the target frame sequence and target text into the first candidate large model to obtain the first candidate review result;

[0120] If the first candidate review result meets the review rules, then the first candidate review result will be used as the target review result;

[0121] If the first candidate review result does not meet the review rules, the model call parameters corresponding to the first candidate large model are adjusted to obtain the second candidate large model, and the target frame sequence and target text are input into the second candidate large model to obtain the second candidate review result;

[0122] If the second candidate audit result meets the audit rules, then the second candidate audit result will be used as the target audit result.

[0123] If the second candidate review result does not meet the review rules, the model call parameters corresponding to the second candidate large model are adjusted to obtain the third candidate large model. The target frame sequence and target text are then input into the third candidate large model to obtain the third candidate review result, which is then used as the target review result.

[0124] In some embodiments, the model call parameters include: temperature parameters, call waiting time, and start / stop status of structured output constraints;

[0125] Adjusting the model invocation parameters corresponding to the initial multimodal large model yields the first candidate large model, including:

[0126] Set the temperature parameters of the initial multimodal large model to zero, set the call wait time to zero, and set the start / stop status of the structured output constraints to on to obtain the first candidate large model;

[0127] Adjusting the model invocation parameters corresponding to the first candidate large model yields the second candidate large model, including:

[0128] Set the temperature parameter of the first candidate large model to the first temperature value, and set the call wait time to the first duration to obtain the second candidate large model; the first temperature value is greater than zero.

[0129] Adjusting the model invocation parameters corresponding to the second candidate large model yields the third candidate large model, including:

[0130] The temperature parameter of the second candidate large model is set to zero, the call waiting time is set to the second duration, and the start / stop status of the structured output constraint is set to off, resulting in the third candidate large model; the duration of the second duration is longer than the duration of the first duration.

[0131] In some embodiments, in order to perform frame extraction processing on the video to be reviewed based on its video duration to obtain a target frame sequence, the video frame extraction module 302 is specifically used for:

[0132] The frame extraction interval of the video to be reviewed is determined based on the video's duration.

[0133] Based on the frame extraction interval, the video to be reviewed is processed by extracting frames to obtain the target frame sequence.

[0134] In some embodiments, in extracting text from the video to be reviewed to obtain target text, the target text extraction module 303 is specifically used for:

[0135] Extract the audio data corresponding to the video to be reviewed;

[0136] The audio data is processed by speech recognition to obtain the initial text;

[0137] Perform character detection on the initial text to identify at least one non-text character in the initial text;

[0138] Delete the non-text characters to obtain the candidate text;

[0139] Based on a preset regular expression, pattern matching is performed on the candidate text to determine at least one injection text among the candidate texts;

[0140] Delete the injected text to obtain the target text.

[0141] In some embodiments, before acquiring the video to be reviewed, the video review device further includes:

[0142] Get the video duration of the initial video;

[0143] When the duration of the initial video is greater than or equal to the duration threshold, the initial video is split into at least two initial sub-videos, and the initial sub-videos are identified as videos to be reviewed.

[0144] The video review device provided in this embodiment of the invention can execute the video review method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the video review method.

[0145] Figure 4 This is a structural schematic diagram of a video review device provided in an embodiment of the present invention.

[0146] like Figure 4 As shown, the video auditing device 400 includes at least one processor 401 and a memory, such as a read-only memory (ROM) 402 and a random access memory (RAM) 403, communicatively connected to the at least one processor 401. The memory stores computer programs executable by the at least one processor. The processor 401 can perform various appropriate actions and processes based on the computer program stored in the ROM 402 or loaded from storage unit 408 into the RAM 403. The RAM 403 can also store various programs and data required for the operation of the video auditing device 400. The processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0147] Multiple components in the video auditing device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, optical disk, etc.; and a communication unit 409, such as a network card, modem, wireless transceiver, etc. The communication unit 409 allows the video auditing device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0148] Processor 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 401 performs the various methods and processes described above, such as video review methods.

[0149] In some embodiments, the video review method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the video review device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by processor 401, one or more steps of the video review method described above may be performed. Alternatively, in other embodiments, processor 401 may be configured to perform the video review method by any other suitable means (e.g., by means of firmware).

[0150] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0153] To provide user interaction, the systems and techniques described herein can be implemented on a video auditing device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the video auditing device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0154] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0155] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0156] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0157] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video review method, characterized in that, The method includes: Get the videos pending review; Based on the video duration of the video to be reviewed, the video to be reviewed is subjected to frame extraction processing to obtain a target frame sequence; the target frame sequence includes at least one target frame; The target text is obtained by extracting text from the video to be reviewed; The target frame sequence and the target text are input into the initial multimodal large model; If the initial multimodal large model has an anomaly in the output process, the parameters of the initial multimodal large model are adjusted to obtain the target multimodal large model, while keeping the target frame sequence unchanged. The target frame sequence and the target text are then input into the target multimodal large model to obtain the target review result.

2. The method according to claim 1, characterized in that, If the initial multimodal large model exhibits anomalies during the output process, the parameters of the initial multimodal large model are adjusted to obtain the target multimodal large model. The target frame sequence remains unchanged, and the target frame sequence and the target text are input into the target multimodal large model to obtain the target review result, including: Adjust the model calling parameters corresponding to the initial multimodal large model to obtain the first candidate large model, and input the target frame sequence and the target text into the first candidate large model to obtain the first candidate review result; If the first candidate review result meets the review rules, then the first candidate review result will be used as the target review result; If the first candidate review result does not meet the review rules, the model call parameters corresponding to the first candidate large model are adjusted to obtain the second candidate large model, and the target frame sequence and the target text are input into the second candidate large model to obtain the second candidate review result; If the second candidate review result meets the review rules, then the second candidate review result will be used as the target review result; If the second candidate review result does not meet the review rules, the model call parameters corresponding to the second candidate large model are adjusted to obtain the third candidate large model. The target frame sequence and the target text are then input into the third candidate large model to obtain the third candidate review result, which is then used as the target review result.

3. The method according to claim 2, characterized in that, The model call parameters include: temperature parameters, call waiting time, and start / stop status of structured output constraints; The step of adjusting the model invocation parameters corresponding to the initial multimodal large model to obtain the first candidate large model includes: Set the temperature parameter of the initial multimodal large model to zero, set the call waiting time to zero, and set the start / stop state of the structured output constraint to on to obtain the first candidate large model; The step of adjusting the model invocation parameters corresponding to the first candidate large model to obtain the second candidate large model includes: Set the temperature parameter of the first candidate large model to a first temperature value, set the call waiting time to a first duration, and obtain the second candidate large model; the first temperature value is greater than zero. The step of adjusting the model invocation parameters corresponding to the second candidate large model to obtain the third candidate large model includes: The temperature parameter of the second candidate large model is set to zero, the call waiting time is set to the second duration, and the start / stop status of the structured output constraint is set to off to obtain the third candidate large model; the duration of the second duration is longer than the duration of the first duration.

4. The method according to claim 1, characterized in that, The step of extracting frames from the video to be reviewed based on its duration to obtain a target frame sequence includes: The frame extraction interval of the video to be reviewed is determined based on the duration of the video to be reviewed. Based on the frame extraction interval, the video to be reviewed is subjected to frame extraction processing to obtain the target frame sequence.

5. The method according to claim 1, characterized in that, The step of extracting text from the video to be reviewed to obtain the target text includes: Extract the audio data corresponding to the video to be reviewed; The audio data is processed by speech recognition to obtain the initial text; Perform character detection on the initial text to identify at least one non-text character in the initial text; Delete the non-text characters to obtain the candidate text; Based on a preset regular expression, pattern matching is performed on the candidate text to determine at least one injection text among the candidate texts; The injected text is deleted to obtain the target text.

6. The method according to claim 1, characterized in that, Before obtaining the video to be reviewed, the following is also included: Get the video duration of the initial video; When the duration of the initial video is greater than or equal to the duration threshold, the initial video is divided into at least two initial sub-videos, and the initial sub-videos are identified as videos to be reviewed.

7. A video review device, characterized in that, include: The data acquisition module is used to acquire videos to be reviewed. The video frame extraction module is used to extract frames from the video to be reviewed based on the video duration to obtain a target frame sequence; the target frame sequence includes at least one target frame. The target text extraction module is used to extract text from the video to be reviewed to obtain the target text. The data input module is used to input the target frame sequence and the target text into the initial multimodal large model; The target review result generation module is used to adjust the parameters of the initial multimodal large model to obtain the target multimodal large model if there is an anomaly in the output result process of the initial multimodal large model, while keeping the target frame sequence unchanged, and inputting the target frame sequence and the target text into the target multimodal large model to obtain the target review result.

8. A video review device, characterized in that, The video review equipment includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video review method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video review method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the video review method according to any one of claims 1-6.

Images