A model training method, a video processing method, an apparatus, a device, and a medium

Abstract

The application discloses a model training method and device, a video processing method and device, equipment and a medium, and applies to the technical field of computers, and comprises the following steps: acquiring training samples with labels and scene data; the training samples are self-recording video data of target objects; a single-modal model and a multi-modal model are used to respectively evaluate the training samples, the training samples are filtered according to evaluation results, and filtered training samples are obtained; the training samples and the scene data are used to perform stage training on a to-be-trained model, the training data and model parameters of a next stage are adjusted by using the training results of a previous stage, until a termination condition is met, and a trained model is obtained. Through single-modal and multi-modal joint evaluation, the quality of single-modal is guaranteed, cross-modal alignment is ensured, and the quality of training data is improved; a stage dynamic adjustment mechanism can optimize training data and parameters in real time according to model performance, fully utilize the training data, and improve model training accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a model training method, a video processing method, an apparatus, a device and a medium. BACKGROUND

[0002] With the rapid development of large language models, the quality of training data has become a core factor determining the pre-training effect of the model. The current mainstream training data screening strategy has obvious limitations. For example, the data screening method based on the CLIP (Contrastive Language-Image Pre-training) model has bias and is prone to misjudgment of hollow text; the method based on aesthetic evaluation is only suitable for single-mode image generation field and is difficult to support the understanding needs of the model for complex scenes; the screening method based on multi-dimensional indicators needs manual specification of indicators and over-reliance on industry experience. In addition, the existing model training directly mixes the training samples, lacks fine-grained analysis and dynamic perception of the data in the training stage, and fails to fully utilize the value of the training samples, resulting in waste of training samples and highlighting the urgency of optimizing data screening and training methods.

[0003] Therefore, how to effectively screen the training data of the model and effectively integrate the training samples in the model training process to improve the accuracy of the model training is a technical problem that needs to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a model training method, a video processing method, an apparatus, a device and a medium, which solve the problem of inaccurate model training caused by poor quality of training data and direct mixing of training data in the prior art.

[0005] To solve the above technical problems, the present application provides a model training method, comprising:

[0006] obtaining a training data set, wherein the training data set includes training samples with labels and scene data; the training samples are self-recording video data of target objects;

[0007] evaluating the training samples by using a single-mode model and a multi-mode model, screening each training sample according to the evaluation results, and obtaining screened training samples;

[0008] training a to-be-trained model by using the screened training samples and the scene data, adjusting the training data and model parameters of the next stage by using the training results of the previous stage until the end condition is met, and obtaining a trained model.

[0009] Optionally, the training sample is evaluated by using a single-modal model and a multi-modal model respectively, including:

[0010] Text in the training sample is extracted and sliced to obtain text segments and time points corresponding to the text segments;

[0011] The audio and video in the training sample are divided based on the time points to obtain audio segments and video segments corresponding to the text segments;

[0012] The single-modal model is used to evaluate the text segments, the audio segments and the video segments respectively to obtain single-modal evaluation results;

[0013] The multi-modal model is used to evaluate the text segments, the audio segments and the video segments to obtain multi-modal evaluation results;

[0014] The evaluation results are obtained based on the single-modal evaluation results and the multi-modal evaluation results.

[0015] Optionally, the single-modal model is used to evaluate the text segments, the audio segments and the video segments respectively to obtain single-modal evaluation results, including:

[0016] A single-modal text model is used to evaluate semantic integrity and grammatical integrity of the text segments to obtain first modal quality values;

[0017] A single-modal speech model is used to evaluate speech intelligibility and speech integrity of the audio segments to obtain second modal quality values;

[0018] A single-modal image model is used to evaluate picture intelligibility and picture composition integrity of the video segments to obtain third modal quality values;

[0019] The single-modal evaluation results are obtained based on the first modal quality values, the second modal quality values and the third modal quality values.

[0020] Optionally, the multi-modal model is used to evaluate the text segments, the audio segments and the video segments to obtain multi-modal evaluation results, including:

[0021] A target text segment, an audio segment corresponding to the target text segment, and a video segment corresponding to the target text segment form a segment group; the target text segment is any one of the text segments;

[0022] The multi-modal model is used to evaluate modal consistency and modal correlation of the segment group to obtain the multi-modal evaluation results;

[0023] The modal consistency includes visual semantic matching degree, role corresponding consistency, semantic scene consistency, text consistency, and character consistency; and the modal correlation includes scene correlation, label consistency, task context correlation, knowledge information density, and scene task correlation.

[0024] Optionally, before obtaining the evaluation result based on the single-modal evaluation result and the multi-modal evaluation result, the method further comprises:

[0025] detecting similarity between the training samples as a redundancy penalty term;

[0026] Correspondingly, obtaining the evaluation result based on the single-modal evaluation result and the multi-modal evaluation result comprises:

[0027] obtaining weights corresponding to the single-modal evaluation result, the multi-modal evaluation result, and the redundancy penalty term;

[0028] obtaining the evaluation result based on the single-modal evaluation result, the multi-modal evaluation result, the redundancy penalty term, and the weights.

[0029] Optionally, the method further comprises: training the to-be-trained model in stages using the screened training samples and the scene data, and adjusting training data and model parameters of a next stage using a training result of a previous stage until a termination condition is met, to obtain a trained model, comprising:

[0030] Step 1: dividing the screened training samples and the scene data into a plurality of subsets according to business fields;

[0031] Step 2: obtaining corresponding data from the plurality of subsets according to a preset proportion as current training data, and training the to-be-trained model using the current training data;

[0032] Step 3: obtaining a current to-be-trained model snapshot when a preset evaluation interval is reached;

[0033] Step 4: evaluating performance of the current to-be-trained model using a multi-task validation set cluster on the current to-be-trained model snapshot to obtain task performance indexes;

[0034] Step 5: adjusting the current training data and adjusting model parameters using a data scheduler based on the task performance indexes;

[0035] Step 6: using the adjusted training data as the current training data, and training the to-be-trained model using the current training data and the adjusted model parameters;

[0036] Step 7: Steps 3 to 6 are repeatedly performed until the end condition is met, and the trained model is obtained.

[0037] The application further provides a video processing method, comprising:

[0038] obtaining a to-be-processed video; the to-be-processed video is a selfie video containing a target object;

[0039] inputting the to-be-processed video into a trained model to obtain an output result; the trained model is a model obtained by using the model training method.

[0040] The application further provides a model training device, comprising:

[0041] a data obtaining module configured to obtain a training data set, wherein the training data set comprises training samples with labels and scene data; the training samples are self-recorded video data of target objects;

[0042] a data screening module configured to evaluate the training samples by using a single-modal model and a multi-modal model respectively, screen the training samples according to evaluation results, and obtain screened training samples;

[0043] a model training module configured to train a to-be-trained model in stages by using the screened training samples and the scene data, adjust training data and model parameters of a next stage by using a training result of a previous stage, until an end condition is met, and obtain a trained model.

[0044] The application further provides a video processing device, comprising:

[0045] a video obtaining module configured to obtain a to-be-processed video; the to-be-processed video is a selfie video containing a target object;

[0046] a video processing module configured to input the to-be-processed video into a trained model to obtain an output result; the trained model is a model obtained by using the model training method.

[0047] The application further provides a device, comprising:

[0048] a memory configured to store a computer program;

[0049] a processor configured to implement the model training method and / or the video processing method when the computer program is executed.

[0050] The application further provides a medium, wherein the medium stores computer executable instructions, and the computer executable instructions are loaded and executed by a processor to implement the model training method and / or the video processing method.

[0051] It can be seen that the application obtains a training data set, the training data set including training samples with labels and scene data; the training samples are self-recording video data of target objects; single-modal models and multi-modal models are used to evaluate the training samples respectively, each training sample is filtered according to the evaluation results to obtain filtered training samples; the filtered training samples and the scene data are used to train a to-be-trained model in stages, and the training data and model parameters of the next stage are adjusted by using the training results of the last stage until the end condition is met, and a trained model is obtained. The single-modal and multi-modal joint evaluation ensures the quality of a single mode (such as video and text) and ensures cross-modal alignment (such as matching the self-recording video data of the target object with the video scene), and solves the problem of insufficient quality evaluation of multi-modal data in the traditional method. In addition, the stage-by-stage dynamic adjustment mechanism can optimize the training data and parameters in real time according to the model performance, avoid invalid samples from occupying resources, improve the utilization efficiency of the training samples and the accuracy of the model training, and make the model more accurately understand the multi-modal information in a specific scene, thereby strengthening the task performance of the model in the specific scene.

[0052] In addition, the application also provides a video processing method, a model training device, a video processing device, equipment and a medium, which also have the above beneficial effects. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0054] Figure 1 A flowchart of a model training method provided for the embodiments of the application;

[0055] Figure 2 A flowchart of a video processing method provided for the embodiments of the application;

[0056] Figure 3 A flowchart of a training data evaluation provided for the embodiments of the application;

[0057] Figure 4 A flowchart of a model training provided for the embodiments of the application;

[0058] Figure 5 A structural schematic diagram of a model training device provided for the embodiments of the application;

[0059] Figure 6 A structural schematic diagram of a video processing device provided for an embodiment of the present application is shown in FIG. 1.

[0060] Figure 7 A structural schematic diagram of a device provided for an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0062] With the advent of the pre-training era of artificial intelligence, the research gradually shifts from the innovation of model architecture to the data-centered research. With the advent of the pre-training model era represented by Bert (Bidirectional Encoder Representations from Transformers, bidirectional encoder representations from transformers), the architecture of general large language models is gradually determined. For the pre-training and fine-tuning tasks of large language models, the ability of large language models ultimately depends on the quality of data, and formulating an effective data screening strategy becomes a decisive factor for the results of model training. Using all original training data for model training is not a good strategy, because the original training data contains a lot of noise, redundant information, bias, and potentially harmful content. The input of a multi-modal model contains the following types: text, language, image (video), etc. The common task of a multi-modal model is modality understanding, dialogue, etc. This means that a multi-modal model not only needs to ensure the quality of data in a single modality (clear images, rich and smooth text, etc.), but also needs to ensure the alignment of visual, textual, and audio modal inputs. For example, a high-quality picture with a paragraph of unrelated text is harmful to training. Currently, in the field of training data screening for model training, the following mainstream strategies are mainly included:

[0063] 1. Filtering training samples in the data set based on CLIP. Filtering the data set based on CLIP is mainly to encode the feature vectors of images and text, and map them in the same latent space. Calculate the similarity between the two, and obtain a high-quality data set of image-text matching by setting a threshold.

[0064] 2、Filter training samples in dataset based on aesthetic evaluation. In generative image tasks, a common strategy is to introduce aesthetic scores by constructing one or more classifiers to score the quality of generated images, such as lighting, composition, color, saturation, etc. These methods mainly improve the correlation between single encoding and visual quality.

[0065] 3、Filter training samples in dataset based on multi-dimensional indicators. The method of filtering data based on multi-dimensional indicators is to evaluate the quality of single modal data and the alignment of multi-modal data, and to integrate the quality of single modal data and the alignment of multi-source data to achieve the purpose of data quality evaluation.

[0066] However, the above methods have some problems:

[0067] (1) CLIP is a quality evaluation method for image-text pairs by calculating the vector similarity of image-text pairs. CLIP model is trained on an unreviewed network data, which inevitably internalizes social bias. For example, it is biased towards Western culture. When using a biased model for data screening, some minority data will be excluded. Secondly, the simple CLIP similarity is a flawed indicator. Studies have shown that some hollow or non-content information texts (texts containing image, picture, and other general words) will get a virtual high similarity score, because the model only captures the modal matching rather than the content alignment.

[0068] (2) The method of filtering dataset based on aesthetic evaluation is mainly applied to the field of image generation, and the evaluation task of the quality of generated images is still a single modal evaluation method in essence.

[0069] (3) The method of filtering dataset based on multi-dimensional indicators manually specifies multiple single modal and multi-modal evaluation indicators and alignment indicators, and relies on industry engineering experience.

[0070] As can be seen, the accuracy of traditional training data screening methods is difficult to guarantee, and it is easy to mis-screen or miss-screen, and it relies on manual experience. Moreover, the current model training method is to directly mix the training data, which lacks fine-grained analysis and processing of data in the model training stage, cannot dynamically perceive the performance of the model in different tasks in the training stage, does not effectively play the role of training data, and causes waste of training data.

[0071] To solve the above problems, the application provides a model training method. In the method, the evaluation and screening of training data are not dependent on rich engineering experience, the quality of single-modal data and the alignment strategy of multi-modal data are comprehensively evaluated, end-to-end multi-modal data screening is realized, full-modal data alignment screening tasks are supported, and in the model training stage, the model performance is evaluated, and the data matching strategy is adjusted in real time, thereby solving the problems of performance decline and data waste caused by uneven data matching in the model training stage.

[0072] For details, please refer to Figure 1 , Figure 1 A flowchart of a model training method provided by the embodiment of the application. The method can include the following steps:

[0073] S101: Obtain a training data set, wherein the training data set includes training samples with labels and scene data; the training samples are self-recording video data of target objects.

[0074] The execution subject of the embodiment is a terminal. The embodiment is not limited to the type of terminal, as long as the terminal can complete the operation of the model training method. It should be noted that the data used for model training in the embodiment includes two types: (1) general scene data (such as the scene data described above): basic fields such as code, mathematics, text (books, web pages), and multi-modal (text and images), which are used to build general knowledge reserves, logical reasoning, and cross-domain understanding ability of the model, and can improve the robustness of the model. (2) targeted scene data (such as the data in the training data set described above): professional data for specific tasks (such as medical treatment, law, and question answering) or vertical fields, which are used to strengthen the accuracy and professionalism of the model in the subdivided scene. The label in the embodiment can be a judgment result, such as a diagnosis result given by a doctor, or a prescription given for the diagnosis result; or it can also be a classification category, such as a disease type given by a doctor. The specific application scenario of the model to be trained can be determined. For example, when a pharmacy issues a prescription drug offline, it needs to confirm the complaint (information to be obtained and cleaned) and the history diagnosis (label) with an online doctor, and the video data obtained by recording this process records the facial video and audio features of the user, etc.

[0075] S102: Evaluate the training samples by using a single-modal model and a multi-modal model respectively, screen the training samples according to the evaluation results, and obtain screened training samples.

[0076] The embodiment is not limited to a single-modal model and a multi-modal model. For example, the single-modal model can be a large language model (LLM), and the multi-modal model can be a visual language model (VLM). The VLM model is a multi-modal artificial intelligence model combining visual (image / video) and language (text) processing capabilities, which can frame the video data into multiple pictures. The embodiment evaluates the training data from different dimensions to train the model with reliable training samples.

[0077] Further, the evaluation of the training samples by the single-modal model and the multi-modal model can include the following steps:

[0078] Step 11: Extracting the text in the training sample and performing slicing processing to obtain text segments and time points corresponding to each text segment;

[0079] Step 12: Dividing the audio and video in the training sample based on the time points to obtain audio segments and video segments corresponding to the text segments;

[0080] Step 13: Performing single-modal evaluation on the text segments, audio segments, and video segments by the single-modal model to obtain single-modal evaluation results;

[0081] Step 14: Performing multi-modal evaluation on the text segments, audio segments, and video segments by the multi-modal model to obtain multi-modal evaluation results;

[0082] Step 15: Obtaining evaluation results based on the single-modal evaluation results and the multi-modal evaluation results.

[0083] The embodiment evaluates the training sample based on the single-modal model and further performs cross-modal evaluation on the training sample by the multi-modal model, which evaluates the quality of the training sample from multiple dimensions to achieve the objective of objective evaluation of data quality. First, the multi-modal data in the video data is separated and processed to cut out the text, audio, and video. The video data is recognized by ASR (Automatic Speech Recognition) to obtain the text content part in the video data. Further, the text can be sliced according to the dialogue content to obtain multiple text segments and time points corresponding to each text segment. The audio and video (which can be understood as picture frames) are divided according to the time points to obtain audio segments and video segments corresponding to the text segments. The single-modal evaluation and multi-modal evaluation are performed based on the divided text segments, audio segments, and video segments to obtain evaluation results.

[0084] Further, the single-modal model is used to perform single-modal evaluation on the text segment, the audio segment and the video segment respectively to obtain a single-modal evaluation result, which can include the following steps:

[0085] Step 131: The semantic integrity and the grammatical integrity of the text segment are evaluated by using a single-modal text model to obtain a first modal quality value.

[0086] Specifically, for the text segment, its semantic integrity and grammatical integrity are evaluated. The semantic integrity can represent whether the text is complete in expression and the semantics are self-consistent (avoiding situations such as sentence breaks, lack of subject, etc.); the grammatical integrity can represent whether there are serious grammatical errors or misuse of punctuation.

[0087] Step 132: The speech intelligibility and the speech integrity of the audio segment are evaluated by using a single-modal speech model to obtain a second modal quality value;

[0088] Specifically, for the audio segment, its speech intelligibility and speech integrity are evaluated. The speech intelligibility can represent the degree of audio affected by background noise, echo, and environmental interference; the speech integrity can represent whether the audio segment is truncated or missing.

[0089] Step 133: The picture intelligibility and the picture composition integrity of the video segment are evaluated by using a single-modal image model to obtain a third modal quality value.

[0090] Specifically, for the video segment, its picture intelligibility and picture composition integrity are evaluated. The picture intelligibility can represent the video resolution and blurriness; the picture composition integrity is used to represent whether the subject in the image is complete or occluded.

[0091] Step 134: Based on the first modal quality value, the second modal quality value and the third modal quality value, a single-modal evaluation result is obtained.

[0092] Specifically, the first modal quality value, the second modal quality value and the third modal quality value jointly constitute the single-modal evaluation result.

[0093] Further, the above-mentioned multi-modal model is used to perform multi-modal evaluation on the text segment, the audio segment and the video segment to obtain a multi-modal evaluation result, which can include the following steps:

[0094] Step 141: The target text segment, the audio segment corresponding to the target text segment, and the video segment corresponding to the target text segment form a segment group; the target text segment is any segment in the text segment;

[0095] Step 142: The modal consistency and the modal correlation of the segment group are evaluated by using a multi-modal model to obtain a multi-modal evaluation result.

[0096] It needs to be explained that the modal consistency includes visual semantic matching degree, role correspondence consistency, semantic scene consistency, text consistency and character consistency. Specifically, the visual semantic matching degree refers to whether the text segment is consistent with the object, action and environment description in the video / image segment; the role correspondence consistency refers to whether the speaker / object mentioned in the text segment matches the character identity in the video / audio segment; the semantic scene consistency refers to whether the semantics described in the audio segment is consistent with the video / image segment (such as discussing medical problems in a hospital environment); the text consistency refers to the consistency degree of the audio / video segment and the ASR transcribed text segment content; and the character consistency refers to whether the character in the video segment is consistent with the role in the text / audio segment.

[0097] It also needs to be explained that the modal correlation includes scene correlation, label consistency, task context correlation, knowledge information density and scene task correlation. Specifically, the scene correlation refers to whether the text segment conforms to the scene semantics of the target field (medical, medical, medical insurance, etc.); the label consistency refers to whether the text segment matches the task label (question and answer, inquiry scene); the task context correlation refers to whether the content in the audio segment is related to the target scene / field (such as medical question and answer); the knowledge information density refers to whether the audio segment contains useful knowledge points (such as medical terms, professional expressions); and the scene task correlation refers to whether the scene in the video segment is related to the target of the downstream task (such as medical scene → medical task).

[0098] Further, before obtaining the evaluation result based on the single-modal evaluation result and the multi-modal evaluation result, the following steps can also be included:

[0099] The similarity between the training samples is detected and used as a redundancy penalty term; accordingly, obtaining the evaluation result based on the single-modal evaluation result and the multi-modal evaluation result can include the following steps: obtaining the weights corresponding to the single-modal evaluation result, the multi-modal evaluation result and the redundancy penalty term; obtaining the evaluation result based on the single-modal evaluation result, the multi-modal evaluation result, the redundancy penalty term and the weights.

[0100] Specifically, the embodiment further adds a redundancy penalty term based on single-modal and multi-modal evaluation. The redundancy penalty term refers to the similarity comparison between the training sample and the previous training sample. The lower the similarity, the more abundant the training sample, the greater the weight, and the higher the similarity, the more repetitive the training sample, the smaller the weight.

[0101] It can be understood that the multi-modal sample of a training sample is represented as follows:

[0102] .

[0103] wherein, xi represents the content (such as image, text, voice) of the i-th modality, and M is the number of modalities.

[0104] Finally, the evaluation result score of the training sample is:

[0105] .

[0106] wherein, represents the quality score of the i-th modality (single-modality quality score); represents the modality weight coefficient, reflecting the importance of different modalities (learned or manually specified); represents the consistency score between multi-modalities (modality consistency); represents the relevance score to the downstream task (modality relevance); represents the redundancy penalty term; , and are hyperparameters for controlling the proportion of each item in the total score.

[0107] S103: Stage training of the to-be-trained model using the screened training samples and scene data, and adjusting the training data and model parameters of the next stage using the training results of the previous stage until the end condition is met, to obtain a trained model.

[0108] In the model training stage, a common strategy is to pre-allocate data proportions such as mathematics, code, and general knowledge, and then mix them directly for model training. This method cannot dynamically perceive the actual model's ability during the training stage. Therefore, the model training strategy provided in this embodiment aims to solve the problems of low training efficiency and resource waste caused by traditional fixed data ratio training methods. The core is to build a dynamic data scheduling closed-loop system based on performance feedback, adjust the training data and model parameters used for training in the next stage based on the training situation of the previous stage, so that the obtained model has higher reliability. It needs to be noted that the to-be-trained model in this embodiment refers to a multi-modal large model.

[0109] Further, the stage training of the to-be-trained model using the screened training samples and scene data, and the adjustment of the training data and model parameters of the next stage using the training results of the previous stage until the end condition is met to obtain a trained model can specifically include the following steps:

[0110] Step 1: divide the screened training samples and scene data into multiple subsets according to business fields;

[0111] Step 2: obtain corresponding data from multiple subsets according to a preset proportion as current training data, and train the to-be-trained model using the current training data;

[0112] Step 3: When a preset evaluation interval is reached, obtain a current to-be-trained model snapshot;

[0113] Step 4: On the current to-be-trained model snapshot, the performance of the current to-be-trained model is evaluated using a multi-task validation set cluster, and each task performance index is obtained.

[0114] Step 5: Based on each task performance index, the data scheduler is used to adjust the current training data and adjust the model parameters.

[0115] Step 6: The adjusted training data is used as the current training data, and the to-be-trained model is trained using the current training data and the adjusted model parameters.

[0116] Step 7: Repeat steps 3 to 6 until the end condition is met, and obtain a trained model.

[0117] Specifically, the to-be-trained model in this embodiment uses a progressive training strategy, which is not fixed in step (step). The training starts with a larger step size, and then evaluates on a specified multi-task validation set cluster. Based on the multi-task evaluation results, the data distribution is dynamically adjusted on the new batch (batch size), and further fine-tuned with a smaller learning rate to improve the model training efficiency.

[0118] Specifically, the filtered training samples and scene data are divided into multiple subsets (such as medical subsets, mathematical subsets, general dialogue subsets, etc.) according to business fields. In the initial training stage, a preset, relatively balanced initial data sampling distribution is used to mix all data, and a high learning rate is configured to start model training. The goal of this stage is to enable the to-be-trained model to quickly learn the general representation and basic ability across modalities. Set an evaluation interval K (such as every T steps or E rounds), and pause the training process. On the current to-be-trained model snapshot (Checkpoint), a comprehensive performance evaluation is performed on a multi-task validation set cluster covering all key capability dimensions, and a set of performance indicators P = {p1, p2,..., pn} is obtained, where pi represents the performance (such as accuracy, BLEU score, etc.) of the model on the ith task or field.

[0119] Step 5 above can include the following steps:

[0120] Data analysis: Calculate the growth slope Δpi of each task performance index relative to the last evaluation, or directly analyze the difference between the absolute value and the target value. The field with slow growth or low absolute value is determined as the short board of the current model's ability.

[0121] Policy decision: the data scheduler dynamically adjusts the data sampling distribution of the next training stage according to the predefined policy (such as: allocating higher sampling weights to the field with the lowest performance growth rate), and continues to train the to-be-trained model using the new data distribution (i.e. the adjusted training data) and the adjusted hyperparameters (i.e. the model parameters). The overall principle followed is to reduce the data sampling weight of the field where the to-be-trained model has performed well, and correspondingly increase the data sampling weight of the short board field, until the performance of the to-be-trained model on all key tasks converges to a satisfactory level or reaches the preset training termination condition.

[0122] As can be seen, in the entire progressive training process, the model parameters and the training data change dynamically. The model parameters are continuously updated through gradient descent, but their optimization trajectory is guided by the dynamic adjustment of the learning rate. In the early stage, a higher learning rate is used for rapid coarse adjustment, and in the later stage, the learning rate is reduced to adapt to the finer data distribution and fine-tune. At the same time, the composition of the training data is not fixed, but forms a closed-loop system based on the real-time performance feedback of the to-be-trained model on multiple tasks, which automatically increases the sampling weight of the data in the "short board" field where the model currently performs poorly, and reduces the proportion of data in the field where the model has mastered. Thus, the intelligent and dynamic allocation of training resources (data) is realized, and the model evolution process is more efficient and balanced.

[0123] The model training method provided by the embodiment of the application can acquire a training data set through S101, wherein the training data set comprises training samples with labels and scene data; the training samples are self-recording video data of target objects; S102, the single-modal model and the multi-modal model are used to evaluate the training samples respectively, the training samples are filtered according to the evaluation results, and filtered training samples are obtained; S103, the filtered training samples and the scene data are used to train a to-be-trained model in stages, and the training data and the model parameters of the next stage are adjusted by using the training results of the last stage until a termination condition is met, and a trained model is obtained. The single-modal and multi-modal joint evaluation ensures the quality of a single mode (such as video, text) and ensures cross-modal alignment (such as matching of self-recording video data of target objects and video scenes), and solves the problem of insufficient quality evaluation of multi-modal data in the traditional method. Moreover, the segmented and progressive dynamic adjustment mechanism can optimize the training data and the parameters in real time according to the model performance, avoids occupation of resources by invalid samples, improves the utilization efficiency of the training samples and the accuracy of the model training, and in combination with the targeted training of the self-recording video data of target objects and the scene data, the model can more accurately understand the multi-modal information in a specific scene, the task performance of the model in the specific scene is strengthened, and the accuracy of model identification / classification is improved.

[0124] For details, please refer to Figure 2 , Figure 2A flowchart of a video processing method provided by an embodiment of the present application. The method can include:

[0125] S201: obtaining a to-be-processed video; the to-be-processed video is a selfie video containing a target object;

[0126] S202: inputting the to-be-processed video into a trained model to obtain an output result; the trained model is a model obtained by using the model training method.

[0127] When the above video processing method can be applied to the question and answer field, for example, a user records a video for himself, describes a complaint, and the video records the facial video and audio features, etc. The trained model is used to comprehensively judge the video, such as extracting text (patient complaints are obtained through ASR), audio (patient audio feature extraction), and video (patient facial detail picture), to give a risk prompt of certain chronic diseases (such as chronic kidney disease and thyroid disease), and whether it is high risk. Moreover, the model can also combine multi-modal information, cross-verify based on different modalities, and has a higher diagnosis accuracy. For example, thyroid disease (hyperthyroidism / hypothyroidism): hyperthyroidism: exophthalmos, facial flushing, and expression hyperactivity; hypothyroidism: pale or waxen face, facial edema, and expression apathy; thyroid enlargement compresses the vocal cords, causing hoarseness or low voice; the voice of a hypothyroid patient is heavy, and the voice of a hyperthyroid patient is sharp. Chronic liver disease: "liver disease face": dull complexion, yellowish skin, and periorbital pigmentation; may be accompanied by spider nevi and telangiectasia. May show low and hoarse voice (due to general weakness and metabolic disorder). In this way, the user can use a hand terminal or a computer terminal to perform early screening of certain chronic diseases such as thyroid disease and chronic liver disease at any time, and can remind the user to go to the hospital for examination in time when the risk is high. The convenience of disease self-checking of the user is improved, and the work pressure of medical workers is reduced.

[0128] The model training method provided by the embodiment of the present application can obtain a to-be-processed video through S201: obtaining a to-be-processed video; the to-be-processed video is a selfie video containing a target object; S202: inputting the to-be-processed video into a trained model to obtain an output result; the trained model is a model obtained by using the model training method. The to-be-processed video is effectively processed by using the trained model, and the processing and analysis ability of the model for the video is improved.

[0129] In order to make the present application more convenient to understand, please refer to Figure 3 and Figure 4 , Figure 3 A flowchart of a training data evaluation method provided by an embodiment of the present application, which can specifically include:

[0130] The ASR transcribes the audio in the video data into text. Since long samples need to be cropped and refined to retain only meaningful parts, for example, in a video of a doctor-patient consultation process, the meaningless segments are removed, and only the patient's complaint part is retained. The specific method is as follows: based on the ASR model with time points, the audio is transcribed into morpheme segments with start time points and end time points; for example: I (1.0s-1.1s) today (1.1s-1.3s) have a stomachache (1.5s-1.6s). With the morphemes and the corresponding start and end time points, the LLM prompt template filters the text segments that meet the following tasks: (1) merging phrases with the same context according to semantics, such as merging: I have a stomachache today (1.0s-1.6s); (2) based on different evaluation dimensions, comprehensive evaluation is carried out based on different large models (single-modal model and multi-modal model). Similarly, based on the time point, the text segment is extracted to obtain the corresponding audio segment and video segment. The single-modal model and the multi-modal model are used to evaluate the text segment, the audio segment and the video segment in multiple dimensions to obtain the quality of the actual video data. For example, modality consistency, modality correlation and redundancy (similarity between training samples). Among them, the modality consistency can include visual semantic matching degree, role correspondence consistency, semantic scene consistency, text consistency and character consistency, and the modality correlation can include scene correlation, label consistency, task context correlation, knowledge information density and scene task correlation. Finally, the weights are summarized. Here, many of the dimensions are based on content understanding dimensions, which are more in-depth and have strong task relevance.

[0131] Figure 4 A model training process example diagram is provided for the embodiments of the present application. Figure 4 The full data in the above formula refers to the training samples and scene data filtered by the above evaluation results. First, 50% of the full data is directly mixed, and the remaining 50% of the data is distinguished by business categories, such as mathematical categories, code categories, etc., and is manually specified. The mixed data is used for training, and then the training effect is evaluated, and the data mixing ratio and model parameters are dynamically adjusted according to the task performance indicators. Through this closed-loop feedback mechanism of training-evaluation-adjustment-retraining, this strategy enables the training process to have dynamic perception and self-optimization capabilities, and can automatically focus computing resources and training attention on the areas that the model needs to improve the most, thereby significantly improving training efficiency and data utilization, and ultimately obtaining a more balanced and powerful multi-modal model.

[0132] The model training device provided by the embodiments of the present application is described below. The model training device described below can be correspondingly referred to the model training method described above.

[0133] For details, please refer to Figure 5 , Figure 5A structural schematic diagram of a model training device provided by an embodiment of the present application can include:

[0134] The data acquisition module 100 is configured to acquire a training data set, wherein the training data set includes training samples with labels and scene data; and the training samples are self-recording video data of target objects.

[0135] The data screening module 200 is configured to evaluate the training samples respectively by using a single-modal model and a multi-modal model, screen the training samples according to evaluation results, and obtain screened training samples.

[0136] The model training module 300 is configured to perform stage-wise training on a to-be-trained model by using the screened training samples and the scene data, adjust training data and model parameters of a next stage by using a training result of a previous stage, until an end condition is met, and obtain a trained model.

[0137] Based on the above embodiment, the data screening module 200 can include:

[0138] The slicing unit is configured to extract text in the training samples and perform slicing processing to obtain text segments and time points corresponding to the text segments.

[0139] The splitting unit is configured to split audio and video in the training samples based on the time points to obtain audio segments and video segments corresponding to the text segments.

[0140] The single-modal evaluation unit is configured to perform single-modal evaluation on the text segments, the audio segments and the video segments respectively by using the single-modal model to obtain single-modal evaluation results.

[0141] The multi-modal evaluation unit is configured to perform multi-modal evaluation on the text segments, the audio segments and the video segments by using the multi-modal model to obtain multi-modal evaluation results.

[0142] The evaluation result acquisition unit is configured to obtain the evaluation results based on the single-modal evaluation results and the multi-modal evaluation results.

[0143] Based on the above embodiment, the single-modal evaluation unit can include:

[0144] The text evaluation sub-unit is configured to evaluate semantic integrity and grammatical integrity of the text segments by using a single-modal text model to obtain first modal quality values.

[0145] The speech evaluation sub-unit is configured to evaluate speech intelligibility and speech integrity of the audio segments by using a single-modal speech model to obtain second modal quality values.

[0146] an image evaluation subunit configured to evaluate picture sharpness and picture composition integrity of the pictures in the video segment by using a single-modal image model to obtain a third-modal quality value;

[0147] a single-modal evaluation result obtaining subunit configured to obtain the single-modal evaluation result based on the first-modal quality value, the second-modal quality value, and the third-modal quality value.

[0148] According to the above embodiment, the multi-modal evaluation unit can comprise:

[0149] a segment group obtaining subunit configured to form a segment group by using a target text segment, an audio segment corresponding to the target text segment, and a video segment corresponding to the target text segment; the target text segment being any one of the text segments;

[0150] an evaluation subunit configured to evaluate modal consistency and modal correlation of the segment group by using the multi-modal model to obtain the multi-modal evaluation result;

[0151] The modal consistency comprises visual semantic matching degree, role correspondence consistency, semantic scene consistency, text consistency, and character consistency; and the modal correlation comprises scene correlation, label consistency, task context correlation, knowledge information density, and scene task correlation.

[0152] According to the above embodiment, the model training device can further comprise:

[0153] a similarity detection module configured to detect similarity between the training samples as a redundancy penalty term before obtaining the evaluation result based on the single-modal evaluation result and the multi-modal evaluation result;

[0154] Correspondingly, the evaluation result obtaining unit can specifically comprise:

[0155] a weight obtaining subunit configured to obtain weights corresponding to the single-modal evaluation result, the multi-modal evaluation result, and the redundancy penalty term;

[0156] an evaluation result obtaining subunit configured to obtain the evaluation result based on the single-modal evaluation result, the multi-modal evaluation result, the redundancy penalty term, and the weights.

[0157] According to any one of the above embodiments, the model training module 300 can comprise:

[0158] a first execution unit configured to perform step 1: dividing the filtered training samples and the scene data into a plurality of subsets according to business fields;

[0159] a second execution unit configured to execute step 2: obtaining corresponding data from the plurality of subsets according to a preset proportion as current training data, and training the to-be-trained model by using the current training data;

[0160] a third execution unit configured to execute step 3: obtaining a current to-be-trained model snapshot when a preset evaluation interval is reached;

[0161] a fourth execution unit configured to execute step 4: evaluating the performance of the current to-be-trained model by using a multi-task validation set cluster on the current to-be-trained model snapshot, and obtaining task performance indicators;

[0162] a fifth execution unit configured to execute step 5: adjusting the current training data by using a data scheduler based on the task performance indicators, and adjusting model parameters;

[0163] a sixth execution unit configured to execute step 6: taking the adjusted training data as the current training data, and training the to-be-trained model by using the current training data and the adjusted model parameters;

[0164] a seventh execution unit configured to execute step 7: repeating steps 3 to 6 until the end condition is met, and obtaining the trained model.

[0165] It should be noted that the order of the modules and units in the above model training device can be changed without affecting the logic.

[0166] The model training device provided by the embodiment of the application is used for obtaining a training data set, wherein the training data set comprises training samples with labels and scene data; the training samples are self-recording video data of target objects; the data filtering module 200 is used for evaluating the training samples by using a single-modal model and a multi-modal model respectively, filtering the training samples according to the evaluation results, and obtaining filtered training samples; the model training module 300 is used for training a to-be-trained model by using the filtered training samples and the scene data in stages, adjusting training data and model parameters of the next stage by using the training results of the last stage until a termination condition is met, and obtaining a trained model. The device solves the problem of insufficient quality evaluation of multi-modal data in the traditional method by single-multi-modal joint evaluation, which guarantees the quality of a single mode (such as video and text) and ensures cross-modal alignment (such as matching of self-recording video data of target objects and video scenes). In addition, the stage-type dynamic adjustment mechanism can optimize the training data and parameters in real time according to the model performance, avoid invalid samples from occupying resources, improve the utilization efficiency of the training samples and the accuracy of the model training, and make the model more accurately understand multi-modal information in a specific scene through targeted training of the self-recording video data of the target objects and the scene data, thereby strengthening the task performance of the model in the specific scene.

[0167] The video processing device provided by the embodiment of the application is described below, and the video processing device described below can be correspondingly referred to the video processing method described above.

[0168] For details, please refer to Figure 6 , Figure 6 The structural schematic diagram of the video processing device provided by the embodiment of the application can include:

[0169] The video acquisition module 400 is used for acquiring a to-be-processed video; the to-be-processed video is a selfie video containing a target object.

[0170] The video processing module 500 is used for inputting the to-be-processed video into the trained model to obtain an output result; the trained model is a model obtained by using the model training method described above.

[0171] It should be noted that the order of the modules and units in the model training device described above can be changed without affecting the logic.

[0172] It should be noted that the video processing device described above can be applied to the medical question and answer field, that is, the application further provides a medical question and answer device, which can include:

[0173] The video acquisition module 400 is specifically configured to acquire a to-be-processed video; the to-be-processed video is a selfie video containing a user complaint;

[0174] The video processing module 500 is specifically configured to input the to-be-processed video into a trained model to obtain an output result, wherein the trained model is a model obtained by using the model training method.

[0175] Correspondingly, the training sample in the model training process is self-recorded video data containing a user complaint.

[0176] The model training device provided in the embodiment of the application comprises a video acquisition module 400 configured to acquire a to-be-processed video; the to-be-processed video is a selfie video containing a target object; and a video processing module 500 configured to input the to-be-processed video into a trained model to obtain an output result; the trained model is a model obtained by using the model training method. In this way, the to-be-processed video is effectively processed by using the trained model, and the processing and analysis capability of the model for the video is improved. Moreover, through the medical question and answer device, a user can perform disease early screening on specific several chronic diseases such as thyroid diseases and chronic liver diseases at any time, and can be reminded to go to a hospital for examination in time when the risk is high. The convenience of disease self-checking of the user is improved, and the work pressure of medical workers is reduced.

[0177] The device provided in the embodiment of the application will be introduced below, and the device described below can be correspondingly referred to the model training method and / or the video processing method described above.

[0178] Please refer to Figure 7 , Figure 7 A structural schematic diagram of a device provided in the embodiment of the application can comprise:

[0179] The memory 10 is configured to store a computer program.

[0180] The processor 20 is configured to execute the computer program to implement the model training method and / or the video processing method.

[0181] The memory 10, the processor 20 and the communication interface 31 can complete mutual communication through the communication bus 32.

[0182] In the embodiment of the application, the memory 10 is configured to store one or more programs, and the program can comprise program code including computer operation instructions. In the embodiment of the application, the memory 10 can store a program for implementing the following functions:

[0183] Acquire a training data set, wherein the training data set comprises training samples with labels and scene data; the training samples are self-recorded video data of a target object.

[0184] The single-modal model and the multi-modal model are used to evaluate the training samples respectively, the training samples are filtered according to the evaluation results, and filtered training samples are obtained;

[0185] The training model is trained in stages by using the filtered training samples and scene data, the training data and model parameters of the next stage are adjusted by using the training results of the previous stage until the end condition is met, and the trained model is obtained;

[0186] And / or;

[0187] Obtain a to-be-processed video; the to-be-processed video is a selfie video containing a target object;

[0188] The to-be-processed video is input into the trained model to obtain an output result; the trained model is a model obtained by using the model training method.

[0189] In a possible implementation, the memory 10 can include a program storage area and a data storage area, where the program storage area can store an operating system, and application programs required by at least one function, and the like; and the data storage area can store data created during use.

[0190] In addition, the memory 10 can include a read-only memory and a random access memory, and provide instructions and data for the processor. A part of the memory can also include an NVRAM. The memory stores an operating system and operation instructions, executable modules or data structures, or a subset of them, or an extended set of them, where the operation instructions can include various operation instructions for implementing various operations. The operating system can include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0191] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices. The processor 20 can be a microprocessor or any conventional processor, etc. The processor 20 can invoke programs stored in the memory 10.

[0192] The communication interface 31 can be an interface of a communication module, used to connect with other devices or systems.

[0193] Of course, it should be noted that, Figure 7 The structures shown do not constitute a limitation on the devices in the embodiments of the present application. In actual applications, the devices can include more or fewer components than those shown, or some components can be combined. Figure 7 The structures shown do not constitute a limitation on the devices in the embodiments of the present application. In actual applications, the devices can include more or fewer components than those shown, or some components can be combined.

[0194] The readable storage medium provided by the embodiments of the present application is described as follows, and the readable storage medium described below can be correspondingly referred to the model training method and / or the video processing method described above.

[0195] The present application also provides a readable storage medium, and the readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the model training method and / or the video processing method described above.

[0196] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0197] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0198] The professional person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present text can be realized by electronic hardware, computer software or combination of both, and in order to clearly show the interchangeability of hardware and software, the composition and steps of each example have been described in the above description. The functions are executed in the form of hardware or software, which depends on the specific application and design constraints of the technical solution. The professional person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0199] Finally, it should be noted that in the present text, the relationship such as first and second belongs to distinguish one entity or operation from another entity or operation, and does not necessarily require or imply any actual relationship or order between the entities or operations. Moreover, the term "includes", "contains" or any other variant is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0200] The model training method, the video processing method, the model training device, the video processing device, the equipment and the medium provided by the present application are described in detail above, the principle and implementation mode of the present application are described in this paper by applying specific examples, and the above description of the examples is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A model training method, characterized in that, The method comprises the following steps: acquiring a training data set, wherein the training data set comprises training samples and scene data with labels; the training samples are self-recording video data of target objects; evaluating the training samples by using a single-modal model and a multi-modal model respectively, screening each training sample according to an evaluation result, and obtaining screened training samples; training a to-be-trained model in stages by using the screened training samples and the scene data, adjusting training data and model parameters in the next stage by using a training result in the last stage, until a termination condition is met, and obtaining a trained model; evaluating the training samples by using a single-modal model and a multi-modal model respectively, comprising the following steps: extracting text in the training samples and performing slicing processing to obtain text segments and time points corresponding to each text segment; segmenting audio and video in the training samples based on the time points to obtain audio segments and video segments corresponding to the text segments; evaluating the text segments, the audio segments and the video segments by using the single-modal model respectively to obtain single-modal evaluation results; evaluating the text segments, the audio segments and the video segments by using the multi-modal model to obtain multi-modal evaluation results; obtaining the evaluation result based on the single-modal evaluation results and the multi-modal evaluation results; before obtaining the evaluation result based on the single-modal evaluation results and the multi-modal evaluation results, the method further comprises the following steps: detecting similarity between the training samples as a redundancy penalty term; correspondingly, obtaining the evaluation result based on the single-modal evaluation results and the multi-modal evaluation results comprises the following steps: obtaining weights corresponding to the single-modal evaluation results, the multi-modal evaluation results and the redundancy penalty term; obtaining the evaluation result based on the single-modal evaluation results, the multi-modal evaluation results, the redundancy penalty term and the weights.

2. The model training method of claim 1, wherein, evaluating the text segments, the audio segments and the video segments by using the single-modal model respectively to obtain single-modal evaluation results comprises the following steps: evaluating semantic integrity and grammatical integrity of the text segments by using a single-modal text model to obtain first-modal quality values; evaluating speech intelligibility and speech integrity of the audio segments by using a single-modal speech model to obtain second-modal quality values; evaluating picture intelligibility and picture composition integrity of the video segments by using a single-modal image model to obtain third-modal quality values; obtaining the single-modal evaluation results based on the first-modal quality values, the second-modal quality values and the third-modal quality values.

3. The model training method of claim 1, wherein, evaluating the text segments, the audio segments and the video segments by using the multi-modal model to obtain multi-modal evaluation results comprises the following steps: forming a segment group by using a target text segment, an audio segment corresponding to the target text segment and a video segment corresponding to the target text segment; the target text segment is any one of the text segments; Evaluate modality consistency and modality correlation of the segment group by using the multi-modal model to obtain the multi-modal evaluation result. The modality consistency includes visual semantic matching degree, role correspondence consistency, semantic scene consistency, text consistency, and character consistency; and the modality correlation includes scene correlation, label consistency, task context correlation, knowledge information density, and scene task correlation. 4.The model training method of any one of claims 1 to 3, wherein, The training method comprises the following steps: Step 1: divide the filtered training samples and the scene data into a plurality of subsets according to business fields; Step 2: obtain corresponding data from the plurality of subsets according to a preset proportion as current training data, and train the to-be-trained model by using the current training data; Step 3: obtain a current to-be-trained model snapshot when a preset evaluation interval is reached; Step 4: evaluate the performance of the current to-be-trained model by using a multi-task validation set cluster on the current to-be-trained model snapshot to obtain task performance indicators; Step 5: adjust the current training data and model parameters based on the task performance indicators by using a data scheduler; Step 6: train the to-be-trained model by using the current training data and the adjusted model parameters; Step 7: repeat steps 3 to 6 until the end condition is met to obtain the trained model.

5. A method for video processing, the method comprising: The method comprises the following steps: Obtain a to-be-input video; The to-be-input video is a selfie video containing a target object; Input the to-be-input video into the trained model to obtain an output result; the trained model is obtained by using the model training method of any one of claims 1 to 4.

6. A model training apparatus characterized by comprising: The method comprises the following steps: A data acquisition module is configured to acquire a training data set, wherein the training data set comprises training samples with labels and scene data; The training samples are self-recorded video data of a target object; A data screening module is configured to evaluate the training samples by using a single-modal model and a multi-modal model respectively, screen the training samples according to evaluation results, and obtain filtered training samples; A model training module is configured to train a to-be-trained model by using the filtered training samples and the scene data in a phased manner, adjust training data and model parameters of a next phase by using a training result of a previous phase, and obtain a trained model until an end condition is met. The data screening module comprises: A slicing unit is configured to extract text in the training samples and perform slicing processing to obtain text segments and time points corresponding to the text segments; A cutting unit is configured to cut audio and video in the training samples based on the time points to obtain audio segments and video segments corresponding to the text segments. The single-modal evaluation unit is configured to perform single-modal evaluation on the text segment, the audio segment and the video segment respectively by using the single-modal model, and obtain single-modal evaluation results; The multi-modal evaluation unit is configured to perform multi-modal evaluation on the text segment, the audio segment and the video segment by using the multi-modal model, and obtain multi-modal evaluation results; The evaluation result acquisition unit is configured to obtain the evaluation results based on the single-modal evaluation results and the multi-modal evaluation results; Further comprising: The similarity detection module is configured to detect the similarity between the training samples before obtaining the evaluation results based on the single-modal evaluation results and the multi-modal evaluation results, and take the similarity as a redundancy penalty term; Correspondingly, the evaluation result acquisition unit comprises: The weight acquisition subunit is configured to acquire weights corresponding to the single-modal evaluation results, the multi-modal evaluation results and the redundancy penalty term; The evaluation result acquisition subunit is configured to obtain the evaluation results based on the single-modal evaluation results, the multi-modal evaluation results, the redundancy penalty term and the weights.

7. A video processing apparatus, comprising: Comprising: The video acquisition module is configured to acquire a to-be-processed video; The to-be-processed video is a selfie video containing a target object; The video processing module is configured to input the to-be-processed video into a trained model, and obtain an output result; the trained model is a model obtained by using the model training method in any one of claims 1 to 4.

8. An apparatus, comprising: Comprising: The memory is configured to store a computer program; The processor is configured to implement the model training method in any one of claims 1 to 4 and / or the video processing method in claim 5 when executing the computer program.

Images