Method, apparatus, device, storage medium and program product for model evaluation

Abstract

Embodiments of the present disclosure provide a method, apparatus, device, storage medium, and program product for model evaluation. The method includes: obtaining a test data set, the test data set includes a test sample, the test sample includes object information for a test object and multiple video clips; using a trained machine learning model, based on the test sample, generating a predicted video clip sequence and its corresponding text description and a segmentation result of the text description, the video clip sequence includes at least two video clips arranged in sequence among multiple video clips; determining the metric values ​​of performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation result of the text description; and determining the model performance of the trained machine learning model based on the metric values ​​of the performance indicator.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, apparatuses, devices, computer-readable storage media, and computer program products for model evaluation. Background Art

[0002] Currently, machine learning models are used in many fields, such as video editing. However, many methods of applying machine learning models to implement video editing are inefficient and the quality of the edited content is low. Therefore, there is a need for an effective way to evaluate machine learning models so that they can be used in efficient and high-quality video editing during the application stage. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for model evaluation is provided. The method includes: obtaining a test data set, the test data set includes a test sample, the test sample includes object information for a test object and multiple video clips; using a trained machine learning model, based on the test sample, generating a predicted video clip sequence and its corresponding text description and a segmentation result of the text description, the video clip sequence includes at least two video clips arranged in sequence among multiple video clips; determining the metric values ​​of performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation result of the text description; and determining the model performance of the trained machine learning model based on the metric values ​​of the performance indicator.

[0004] In a second aspect of the present disclosure, a device for model evaluation is provided. The device includes: an acquisition module configured to acquire a test data set, the test data set includes a test sample, the test sample includes object information for a test object and multiple video clips; a generation module configured to generate a predicted video clip sequence and its corresponding text description and a segmentation result of the text description based on the test sample using a trained machine learning model, the video clip sequence includes at least two video clips arranged in sequence among multiple video clips; a first determination module configured to determine the metric values ​​of performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation result of the text description; and a second determination module configured to determine the model performance of the trained machine learning model based on the metric values ​​of the performance indicator.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1A A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 1B A schematic diagram showing another example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 A flowchart showing a process for model evaluation during a testing phase according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A schematic diagram showing an example architecture for utilizing a machine learning model for video generation at the inference stage according to some embodiments of the present disclosure;

[0014] Figure 4 A flowchart showing a process of machine learning model training according to some embodiments of the present disclosure is shown;

[0015] Figure 5 An exemplary structural block diagram of an apparatus for model evaluation according to some embodiments of the present disclosure is shown; and

[0016] Figure 6 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0020] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0024] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0025] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0026] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0027] Figure 1AA schematic diagram of an example environment 100 in which an embodiment of the present disclosure can be implemented is shown. In the environment 100, an electronic device 110 applies a machine learning model 105 to perform video editing in a testing phase. The machine learning model 105 is configured to generate a test file 114. The machine learning model 105 is configured to process a prompt word 112 input by a user to generate a test file 114. The prompt word 112 can be a text prompt word to describe the task to be performed. The machine learning model 105 can be configured to perform a video editing task, and its input can include a text description of the video editing task and a video material 103 to be edited. In some embodiments, the prompt word 112 is used to guide the machine learning model 105 to generate a test file 114 for a test object. The test file 114 may include video-related information. In the application phase, the electronic device 110 can generate a target video using the trained machine learning model 105.

[0028] In environment 100, electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, gaming device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of interface for the user (such as "wearable" circuit, etc.). Visual generation model 105 can be implemented, for example, in various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0029] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0030] Figure 1B 1 shows a schematic diagram of an example environment 102 in which embodiments of the present disclosure can be implemented. Figure 1B In the environment 102, it is desirable to train and use such a machine learning model (i.e., machine learning model 130), which is configured for a variety of application environments. For example, the machine learning model 130 can be configured to perform a video editing task, and its input can include prompt words for the video editing task and the video material to be edited.

[0031] like Figure 1BAs shown, the environment 102 includes a model training system 150, a model testing system 160, and a model application system 170. In some embodiments, the trained machine learning model 130 can be directly provided to the model application system 170 for use, or provided to the model testing system 160 for testing and to the model application system 170 for use at the same time.

[0032] Before training, the parameter values ​​of the machine learning model 130 may have initial values, or may have pre-trained parameter values ​​obtained through a pre-training process. The machine learning model 130 may be trained via forward propagation and back propagation, and the parameter values ​​of the machine learning model 130 may be updated and adjusted during the training process. The machine learning model 130' may be obtained after the training is completed. At this point, the parameter values ​​of the machine learning model 130' have been updated, and based on the updated parameter values, the machine learning model 130 may be used to implement the recommendation task in the model application stage.

[0033] In the model training phase, the model training system 150 can train the machine learning model 130 based on a training data set 152 including a plurality of training samples 154. The training samples 154 including sample inputs 155 and true value outputs 156 can be used to train the machine learning model 130. Specifically, the training process can be iteratively performed using a large number of training samples. After the training is completed, the machine learning model 130 may include knowledge about the task to be processed.

[0034] During the model testing phase, the model testing system 160 may use a test data set 162 to test the trained machine learning model 130'. The test data set 162 is similar to the training data set 152 and also includes a plurality of test samples 164, wherein the test sample 164 may include a sample input 165 and a true value output 166. The model testing system 160 uses the test data set 162 to evaluate the various performance indicators of the trained machine learning model 130' to determine whether the model performance of the trained machine learning model 130' meets expectations.

[0035] In the model application stage, the trained machine learning model 130' (at this time, the machine learning model 130' has trained parameter values) can be used to perform the corresponding task. The model application system 170 can use the trained machine learning model 130' to determine the model output 144 based on the model input 172. In the scenario of the video editing task, the model input includes a text description of the video editing task and a plurality of video clips to be edited. The model output can include a clipped target file for generating a clipped video.

[0036] exist Figure 1BIn the figure, the model training system 150, the model testing system 160 and / or the model application system 170 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device may involve any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Servers include but are not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like. Although shown as separate systems in the figure, the model training system 150, the model testing system 160 and / or the model application system 170 can be integrated into a single device / system, and the embodiments of the present disclosure are not limited in this respect.

[0037] Currently, machine learning models can be applied to the field of video editing, such as video advertising. Generally speaking, an excellent video mix usually contains a logically ordered sequence of clips, attractive spoken text (also known as text description), a strong correlation between the spoken text and the corresponding video clips, appropriate subtitles, etc. However, a lot of effort is required to manually design video ideas, including editing and sorting, writing spoken text, and ensuring its logical coherence. This complexity often poses significant challenges to users. Therefore, a new evaluation scheme for machine learning models used in the field of video editing is needed.

[0038] In order to solve at least some of the above problems, in an embodiment of the present disclosure, a scheme for model evaluation is proposed. In this scheme, a test data set is obtained, and the test data set includes test samples, and the test samples include object information for a test object and multiple video clips. Using a trained machine learning model, based on the test samples, a predicted video clip sequence and its corresponding text description and a segmentation result of the text description are generated, and the video clip sequence includes at least two video clips arranged in sequence from multiple video clips. Determine the metric values ​​of the performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation result of the text description. Based on the metric values ​​of the performance indicators, determine the model performance of the trained machine learning model.

[0039] By implementing the evaluation of the machine learning model according to the scheme disclosed in the present invention, the machine learning model with good performance after testing can improve the efficiency of video editing and enhance the quality of the finished film during the application stage.

[0040] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0041] Figure 2The flowchart of process 200 for evaluating a machine learning model in a test phase according to some embodiments of the present disclosure is shown. Process 200 can be implemented in Figure 1A electronic device 110 or Figure 1B model test system 160.

[0042] In block 210, electronic device 110 obtains a test data set, which includes test samples. The test samples include object information for a test object and multiple video segments.

[0043] In some embodiments, the type and number of test objects can be set according to different video application fields or application scenarios. For example, in the field of video advertising, the test object can be a certain product (or commodity), or multiple different products of a certain type of product, or multiple types of products, etc., which are not limited here. Assuming the test object is a certain mobile phone, then the multiple video segments can be, for example, multiple different introduction videos for this mobile phone, and the object information can be, for example, the name or advertisement text of this mobile phone, etc., which are not limited here.

[0044] In block 220, further, electronic device 110 uses the trained machine learning model 105 to generate a predicted video segment sequence and its corresponding text description and the segmentation result of the text description based on the test samples. The video segment sequence includes at least two video segments arranged in sequence among the multiple video segments.

[0045] When using the trained machine learning model 105 to generate a predicted video segment sequence and its corresponding text description and the segmentation result of the text description based on the test samples, a test file 114 can be generated based on the multiple video segments and test prompt words. The test prompt words can at least include the object information of the test object and the identification information of each of the multiple video segments. The test file 114 can include the identification information of the predicted video segment sequence, as well as the predicted text description and the segmentation result of the text description. Then, the predicted video segment sequence can be obtained based on the identification information of the predicted video segment sequence.

[0046] In block 230, further, electronic device 110 determines the measurement values of the performance metrics corresponding to the predicted video segment sequence, text description, and the segmentation result of the text description, respectively. How to determine the measurement values of the performance metrics corresponding to the predicted video segment sequence, text description, and the segmentation result of the text description, respectively, will be discussed in detail below.

[0047] In some embodiments, when determining the metric values of the performance metrics corresponding to the predicted video clip sequence, the text description, and the segmentation result of the text description respectively, the electronic device 110 may utilize another machine learning model to generate the metric values of the performance metrics corresponding to at least one of the predicted video clip sequence, the text description, and the segmentation result of the text description based on the scoring requirements for the performance metrics corresponding to at least one of the video clip sequence, the text description, and the segmentation result of the text description.

[0048] In such an embodiment, the another machine learning model may be constructed based on a language model. The another machine learning model may be configured to receive a prompt for model evaluation and generate corresponding multiple metric values based on the scoring requirements for multiple performance metrics in the prompt. It should be understood that when evaluating the predicted video clip sequence, the text description, and the segmentation result of the text description respectively, the another machine learning model may be used for evaluation respectively, or other methods may be used for evaluation.

[0049] In some embodiments, when determining the metric value of the performance metric corresponding to the predicted video clip sequence, the electronic device 110 may determine the metric values of the corresponding overlap performance metric and accuracy performance metric. It should be understood that in other embodiments, other performance metrics for the predicted video clip sequence may also be used for evaluation.

[0050] In some embodiments, when determining the metric value of the overlap performance metric corresponding to the predicted video clip sequence, based on the intersection of the video clips included between the predicted video clip sequence and the ground-truth video clip sequence corresponding to the test sample, the electronic device 110 may determine the metric value of the overlap performance metric. As an example, the larger the intersection of the video clips of the two, the larger the metric value of the overlap performance metric corresponding to the predicted video clip sequence. This means that if the video clips in the predicted video clip sequence are exactly the same as the video clips in the ground-truth video clip sequence, that is, the two are completely overlapped, the metric value of the overlap performance metric is the largest.

[0051] In some embodiments, when determining the metric value of the accuracy performance metric corresponding to the predicted video clip sequence, based on whether each video clip arranged in sequence in the predicted video clip sequence is the same as that in the ground-truth video clip sequence, the electronic device 110 may determine the metric value of the accuracy performance metric. As an example, in the case where the video clips of the two are the same, if the arrangement order of the video clips is also the same, it means that the accuracy is higher, that is, the metric value of the accuracy performance metric is larger. If the video clips of the two are exactly the same and the arrangement order is also exactly the same, the metric value of the accuracy performance metric can reach the maximum.

[0052] By using the overlap index and the accuracy index to evaluate the predicted video clip sequence, the quality of the subsequent film can be improved.

[0053] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, another machine learning model can be used. As an example, another machine learning model can perform an evaluation based on the input prompt words for the text description evaluation. The prompt words may include evaluation requirements and scoring requirements for each performance indicator. For example, the evaluation requirement may be "You need to evaluate the text description based on the following performance indicators: authenticity index, text fluency index, contextual coherence index, logic index, picture relevance index, object relevance index". It should be understood that any other appropriate performance indicators can also be used to evaluate the text description.

[0054] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the authenticity indicator corresponding to the predicted text description.

[0055] Specifically, it can be determined whether the predicted text description is in fact consistent with the corresponding video clip and object information. In other words, the authenticity index can be used to determine whether the predicted text description is true or fictitious relative to the corresponding video clip and object information. For example, if price information appears in the text description, but the price information cannot be verified from the corresponding object information or video clip, it can be determined that the authenticity of the predicted text description is low, that is, the corresponding authenticity index has a low metric value.

[0056] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the text fluency indicator corresponding to the predicted text description. Text fluency may refer to text coherence.

[0057] Specifically, it is possible to determine whether the text description is fluent, whether there are typos, whether it is easy to read aloud, etc., and set scoring intervals corresponding to different fluency levels to determine the measurement value of the text description with respect to the text fluency index.

[0058] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the context coherence indicator corresponding to the predicted text description.

[0059] Specifically, it can determine whether the text is contradictory, whether it repeats large sections, whether it is not concise enough, whether it is semantically connected, etc., and set scoring ranges corresponding to different degrees of coherence to determine the measurement value of the text description regarding the contextual coherence indicator.

[0060] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the logic indicator corresponding to the predicted text description.

[0061] Specifically, it is possible to determine whether the predicted text description has logical errors, whether it is logically connected, etc., so as to determine the metric value of the text description with respect to the logic index. For example, for the text description "The lens is coated with an oleophobic layer to prevent oil stains", this is logical and can be evaluated with a higher metric value. For another example, for the text description "Tea brewed in summer cools faster than tea brewed in winter", this is unreasonable and can be evaluated with a lower metric value.

[0062] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the correlation indicator between the predicted text description and the video frames in the predicted video clip sequence. In such an embodiment, some video frames need to be extracted from the predicted video clip sequence for calculating the metric value of the picture correlation indicator.

[0063] Specifically, it is possible to determine whether the text description corresponding to the picture content of the video frame is related to the picture content, or whether the two are highly consistent, or whether the two are unrelated to each other, etc., and set different scores for different degrees of relevance, thereby determining the metric value of the correlation index between the predicted text description and the video frame in the predicted video clip sequence.

[0064] Alternatively or additionally, in some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the correlation indicator between the predicted text description and the object information of the test object.

[0065] Specifically, it is possible to determine whether the predicted text description is related to the object information of the test object, or whether the test object is described in detail from multiple aspects, or whether the two are unrelated to each other, etc., and set different scores for different degrees of relevance, thereby determining the metric value of the correlation index between the predicted text description and the object information of the test object.

[0066] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted text description, the electronic device 110 may determine the metric value of the character length indicator of the text description based on the difference between the target number of characters of the predicted video clip sequence and the number of characters contained in the text description.

[0067] In such an embodiment, the duration of the corresponding audio can be determined based on the duration of the predicted video clip sequence, and then the target number of characters of the predicted video clip sequence can be determined. As an example, if the difference between the target number of characters of the predicted video clip sequence and the number of characters contained in the text description is smaller, or the target number of characters is closer to the number of characters contained in the text description, it can be determined that the metric value of the character length index of the predicted text description is larger. Conversely, if the difference between the target number of characters of the predicted video clip sequence and the number of characters contained in the text description is larger, the metric value of the character length index of the predicted text description can be smaller.

[0068] In the above embodiment, the predicted text description is evaluated based on authenticity index, text fluency index, contextual coherence index, logic index, screen relevance index, object relevance index, and character length index, which is beneficial to improving the quality of the model in the application stage.

[0069] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted segmentation result of the text description, based on whether the segmentation result of the text description meets at least one segmentation requirement, the electronic device 110 may determine the metric value of the performance indicator corresponding to the segmentation result. For ease of understanding, the segmentation requirement may correspond to a display requirement for subtitles on a screen of a video frame.

[0070] In some embodiments, at least one segmentation requirement may indicate a requirement for the number of characters of a text segmentation unit corresponding to each video frame in the video segment sequence. That is, when displaying subtitles, the text content of a text segmentation unit may be displayed on the screen of each video frame. The number of characters of each text segmentation unit may be required to be within a preset character number range.

[0071] Alternatively or additionally, in some embodiments, at least one segmentation requirement may indicate a requirement on the number of text segmentation units into which the text content between specified punctuation marks of the text description is divided. The specified punctuation marks may include, for example, a period, a comma, or other punctuation marks, etc., which are not limited here. For example, each sentence in the text description may be required to be segmented into several text segmentation units according to the period.

[0072] Alternatively or additionally, in some embodiments, at least one segmentation requirement may indicate a segmentation requirement for partial words in the text description. For example, it may be required that the partial words cannot be cut off, that is, must be in the same text segmentation unit. As an example, such partial words may be commonly used basic vocabulary.

[0073] In some embodiments, when determining the metric value of the performance indicator corresponding to the predicted segmentation result of the text description, the corresponding metric value can be determined for each of the above segmentation requirements. In some embodiments, when the predicted text description meets multiple segmentation requirements at the same time, it can be determined that its metric value is larger. Correspondingly, the fewer segmentation requirements are met, the smaller the metric value can be.

[0074] By evaluating the predicted segmentation results of the text description with respect to the above-mentioned multiple segmentation requirements, the readability of the subtitles in the subsequent film can be improved.

[0075] In block 240, further, the electronic device 110 determines the model performance of the trained machine learning model 105 based on the metric value of the performance indicator. Thus, the model performance of the machine learning model 105 can be determined by evaluating the multiple performance indicators discussed above for the predicted video clip sequence, text description, and the segmentation results of the text description.

[0076] It should be understood that if the performance meets expectations, the machine learning model 105 can be put into use for reasoning (or application). If the performance does not meet expectations, it is possible to continue to return to the model training phase and update the machine learning model 105 until the evaluated model performance meets the expected performance.

[0077] The evaluation process of the machine learning model is beneficial to improving the quality of subsequent models in the application stage.

[0078] Figure 3 FIG. 3 is a schematic diagram of an example architecture 300 for utilizing a trained machine learning model for video generation at the inference stage according to some embodiments of the present disclosure. For ease of discussion, reference will be made to Figure 1A These embodiments are described with reference to the environment 100 of FIG. These embodiments may be implemented in Figure 1A electronic device 110 or Figure 1B The model is applied in system 170.

[0079] like Figure 3 As shown, in the architecture 300, the electronic device 110 obtains multiple video clips 310 for the target object and object information 320 of the target object. In some embodiments, the type and number of target objects can be set according to different video application fields or application scenarios. For example, in the field of video advertising, the target object can be a certain product (or commodity), or multiple different products of a certain type of product, or multiple types of products, etc., which are not limited here. Assuming that the target object is a certain mobile phone, the multiple video clips 310 can be, for example, multiple different introduction videos for the mobile phone, and the object information 320 can be, for example, the name of the mobile phone or advertising text, etc., which are not limited here.

[0080] In some embodiments, when acquiring multiple video clips 310, the electronic device 110 may determine to select multiple video clips 310 that match the object information 320 from the video material library 330. As an example, the video material library 330 may include one or more video clips corresponding to multiple objects. When acquiring multiple video clips 310 for a target object, the video material library 330 may be queried for multiple video clips for the target object, and then the queried multiple video clips 310 may be acquired.

[0081] It should be noted that the obtained multiple video clips 310 for the target object may be out of order. In addition, the content in the video material library 330 may be updated, for example, new video clips may be added, old video clips that do not meet the requirements may be deleted, etc., and video clips corresponding to new objects may be added, and video clips corresponding to original objects may be deleted.

[0082] Further, the electronic device 110 generates a target file 314 based on multiple video clips 310 and prompt words 312 using the trained machine learning model 105. As an example, the trained machine learning model 105 can be constructed based on a language model (LM). The machine learning model 105 can be configured to perform a video editing task based on multimodal input. The video editing task may include a video selection task, a text description generation task of a text description corresponding to the video (i.e., a spoken copy), and a text description segmentation task for subtitles.

[0083] In some embodiments, the data format of the target file 314 can be any suitable type of data format, such as JSON data format or other data formats. The JSON protocol can include information for the video track 352, the audio track 354, and the subtitle track 356.

[0084] Here, the plurality of video segments 310 respectively have corresponding first identification information. The prompt word 312 includes at least object information and the first identification information of each of the plurality of video segments 310. As an example, the first identification information of each video segment may include the ID number of the video segment. As another example, the first identification information of each video segment may include resource path information of the video segment, such as web page path information, etc. It should be understood that the first identification information of the video segment may be any appropriate information for identifying the video segment or being able to locate the position of the video segment.

[0085] In some embodiments, the prompt word 312 may also include a number requirement of the video segments to be selected. According to the number requirement, the machine learning model 105 may be used to select the video segments of the number from the plurality of video segments 310 for video editing.

[0086] Alternatively or additionally, in some embodiments, the prompt word 312 may also include a character count requirement for the text description to be generated. In order to configure audio with a duration matching the selected video segment, the character count of the text description may be limited so that the audio converted from the text description can be aligned with the selected video segment in duration.

[0087] Alternatively or additionally, in some embodiments, the prompt word 312 may also include a segmentation requirement for the text description. The segmentation requirement may correspond to a display requirement for subtitles. In some embodiments, the segmentation requirement for the text description may at least indicate the character number range of the text segmentation unit corresponding to each video frame in the video clip. That is, when displaying subtitles, the text content of a text segmentation unit may be displayed on the screen of each video frame, and the number of characters of the text content may be within a preset range. In addition, the segmentation requirement may also indicate how many text segmentation units each sentence in the text description may be segmented into, and may also indicate which words cannot be cut off, i.e., must be in the same text segmentation unit, and so on.

[0088] In addition, the prompt may also include a description of the video editing requirements. Such a description may be, for example, "You are an advertising video creator, and now you want to create an advertising video that attracts viewers in the following way: naturally connect some video clips, add advertising explanations, and segment the oral explanations to obtain sentence results to make subtitles", etc.

[0089] Alternatively or additionally, in some embodiments, the prompt word 312 may also include example generated content for determining the format of generated content in the target file 314. By providing the example generated content to the machine learning model 105 for reference, the format of the content generated by the machine learning model 105 may be the same as the format of the example generated content. This can facilitate the understanding of the content in the target file 314.

[0090] The target file 314 includes at least first identification information corresponding to at least one video segment arranged in sequence among the multiple video segments 310, a text description of the at least one video segment, and a segmentation result of the text description. Further, the electronic device 110 generates a target video for the target object based at least on the target file 314. In this way, at least one video segment in the generated target video can be at least one video segment arranged in sequence selected from the multiple unordered video segments 310. It should be understood that in some cases, only one video segment can be selected from the multiple video segments 310 for editing.

[0091] In some embodiments, when generating a target video, the electronic device 110 may obtain at least one video segment based on the first identification information corresponding to the at least one video segment in the target file 314. Then, based on the text-to-speech processing, the electronic device 110 may convert the text description into audio. As an example, the electronic device 110 may convert the text description into audio with the help of a text-to-speech tool. Then, the electronic device 110 may match the at least one video segment arranged in sequence with the audio and the segmentation result of the text description, respectively, to generate the target video.

[0092] In some embodiments, the target file 314 also includes the second identification information of the audio corresponding to the text description. As an example, the second identification information of the audio can include the ID number of the audio. As another example, the second identification information of the audio can include the resource path information of the audio. It should be understood that the second identification information of the audio can be any suitable information that can identify the audio.

[0093] In some embodiments, when generating a target video, based on the second identification information, the electronic device 110 can obtain audio corresponding to the text description, and then can match at least one video segment arranged in sequence with the audio and the segmentation results of the text description to generate the target video.

[0094] In some embodiments, when matching at least one video segment with the audio and the segmentation result of the text description, the electronic device 110 can at least determine the first duration of the at least one video segment and determine the second duration of the audio based on the number of characters in the text description. Then, the electronic device 110 can adjust the number of characters in the text description and / or adjust the second duration of the audio to match the first duration.

[0095] Specifically, the number of characters in the text description to be adjusted and / or the second duration of the audio to be adjusted can be determined based on the difference between the second duration and the first duration. For example, if the duration of the audio is much longer than the duration of at least one video clip, the text description can be streamlined to reduce the number of characters, thereby making the converted audio duration shorter. If the second duration is only slightly longer than the first duration, the speech rate of the audio can be increased to match the first duration. The reverse is also similar and will not be described in detail here. This helps to improve the matching efficiency of the audio duration to the duration of at least one video clip.

[0096] In some embodiments, when matching at least one video segment with the audio and the segmentation results of the text description, the electronic device 110 may also segment the audio into at least one audio segment based on the segmentation results of the text description, and align at least one video segment with at least one audio segment and the segmentation results of the text description in time to generate a target video. In such an embodiment, the video segments arranged in sequence may be aligned in time with the corresponding audio segments and the corresponding one or more text segmentation units, so that at least one video segment is matched with the audio and the segmentation results of the text description. This ensures the accuracy of the matching, thereby improving the quality of the generated target video.

[0097] Through the embodiments of the present disclosure, a trained machine learning model is used to generate a target file including identification information corresponding to the video clips, corresponding text descriptions, and segmentation results of the text descriptions based on multiple video clips and prompt words, and a target video for a target object is generated at least based on the target file, which can improve the efficiency of video editing and enhance the quality of the finished film.

[0098] Figure 4 FIG. 4 is a flowchart showing a process 400 of machine learning model training according to some embodiments of the present disclosure. Figure 1A These embodiments are described with reference to the environment 100 of FIG. These embodiments may be implemented in Figure 1A electronic device 110 or Figure 1B In the model training system 150.

[0099] like Figure 4 As shown, in box 410, the electronic device 110 obtains a training data set, and the training data set includes at least a first training sample and first annotation information corresponding to the first training sample. The first training sample includes first object information for a first sample object and multiple video clips. The first annotation information includes a true value video clip sequence for the first sample object, a corresponding first true value text description, and a segmentation result of the first true value text description.

[0100] The annotation information discussed herein may be the true value information corresponding to the training sample, and the true value information may be included in the training data set. Such a training data set may be used for the machine learning model 105 to be trained to perform the video editing task. The video editing task may include a video selection task, a text description generation task of the text description corresponding to the video (i.e., the spoken copy), and a text description segmentation task for subtitles.

[0101] The training samples discussed herein may include a large number of sample objects. In some embodiments, the type and number of individual sample objects in the sample objects may be set according to different video application fields or application scenarios. For example, in the field of video advertising, a single sample object may be a certain product (or commodity), or may be a plurality of different products of a certain type of product, or may be a plurality of types of products, and so on, without limitation here. Assuming that a single sample object is a certain mobile phone, the corresponding plurality of video clips may be, for example, a plurality of different introduction videos for the mobile phone, and the corresponding sample object information may be, for example, the name of the mobile phone or an advertisement text, and so on, without limitation here.

[0102] It should be noted that the multiple video clips in the first training sample may be multiple unordered video clips. The video clips in the true value video clip sequence may be ordered video clips, that is, video clips arranged in sequence. The machine learning model 105 may be configured to perform a video editing task based on multimodal input.

[0103] In block 420, further, the electronic device 110 performs a first training task on the model based on the training data set in the following manner: The first training task may be referred to as a composite training task.

[0104] Specifically, the electronic device 110 uses the machine learning model 105 to be trained to generate a prediction result for the first sample object based on the first training sample, and the prediction result includes a first predicted video clip sequence, a corresponding first predicted text description, and a segmentation result of the first predicted text description.

[0105] When the machine learning model 105 to be trained is used to generate a first predicted video segment sequence, a corresponding first predicted text description, and a segmentation result of the first predicted text description based on the first training sample, a training intermediate file 114 can be generated based on multiple video segments and training prompt words in the first training sample. The training prompt words can at least include object information of the first sample object and identification information of each of the multiple video segments. The training intermediate file 114 can include identification information of the first predicted video segment sequence, as well as the first predicted text description and the segmentation result of the first predicted text description. Then, the first predicted video segment sequence can be obtained based on the identification information of the first predicted video segment sequence.

[0106] Further, the electronic device 110 trains the machine learning model 105 based on the difference between the prediction result and the first annotation information. The training goal of the machine learning model 105 to be trained is to make the prediction result output by the machine learning model 105 as close as possible to the first annotation information. Therefore, during training, a corresponding training loss can be defined based on the difference between the prediction result and the first annotation information. The training loss can be expressed as any appropriate loss function that can determine the difference between the prediction result and the first annotation information. The difference between the prediction result and the first annotation information can be minimized or reduced to meet the predetermined goal. Then, the trained machine learning model 105 can be used Figure 2 in the testing phase.

[0107] In some embodiments, the electronic device 110 may perform pre-training on the model by executing a second training task on the machine learning model 105. The second training task may be referred to as a video mixing task for a video track.

[0108] The execution of the second training task may include: the electronic device 110 may select a second training sample and a second annotation information from the training data set. The second training sample may include the second object information of the second sample object and a plurality of unordered video clips. The plurality of video clips may include at least one video clip matching the second object information and at least one video clip unrelated to the second object information. The second annotation information may include an ordered sequence of video clips matching the second object information. Using the machine learning model 105 to be trained, a second predicted video clip sequence may be generated based on the second training sample. Based on the difference between the second predicted video clip sequence and the ordered video clip sequence, the machine learning model 105 may be trained.

[0109] In the pre-training stage of this video remix task, the training goal of the machine learning model 105 to be trained is to make the second predicted video segment sequence predicted by the machine learning model 105 as close as possible to the true ordered video segment sequence. Therefore, in this pre-training stage, the corresponding training loss can be defined based on the difference between the second predicted video segment sequence and the ordered video segment sequence. The training loss can be expressed as any appropriate loss function that can judge the difference between the second predicted video segment sequence and the ordered video segment sequence. The difference between the second predicted video segment sequence and the ordered video segment sequence can be minimized or reduced to meet the corresponding predetermined goal.

[0110] In some embodiments, the electronic device 110 may perform pre-training on the model by performing a third training task on the model. The third training task may be referred to as a text prediction task for an audio track.

[0111] The execution of the third training task may include: the electronic device 110 may select a third training sample and third annotation information from the training data set. The third training sample may include third object information of the third sample object and a video clip sequence matching the third object information. The third annotation information may include a second true value text description corresponding to the video clip sequence. Using the machine learning model 105 to be trained, a second predicted text description may be generated based on the third training sample. Based on the difference between the second predicted text description and the second true value text description, the machine learning model 105 may be trained.

[0112] In the pre-training stage of this text prediction task, the training goal of the machine learning model 105 to be trained is to make the second predicted text description predicted by the machine learning model 105 as close as possible to the second true text description. Therefore, in this pre-training stage, the corresponding training loss can be defined based on the difference between the second predicted text description and the second true text description. The training loss can be expressed as any appropriate loss function that can judge the difference between the second predicted text description and the second true text description. The difference between the second predicted text description and the second true text description can be minimized or reduced to meet the corresponding predetermined goal.

[0113] In some embodiments, the electronic device 110 may perform training on the model by performing a fourth training task on the model. The fourth training task may be referred to as a text segmentation task for a subtitle track.

[0114] The execution of the fourth training task may include: the electronic device 110 may select a fourth training sample and fourth annotation information from the training data set. The fourth training sample may include fourth object information of a fourth sample object and a video clip sequence matching the fourth object information. The fourth annotation information may include a segmentation result of a third true value text description corresponding to the video clip sequence. Using the machine learning model 105 to be trained, a segmentation result of a third predicted text description may be generated based on the fourth training sample. Based on the difference between the segmentation result of the third predicted text description and the segmentation result of the third true value text description, the machine learning model 105 may be trained.

[0115] In the training phase of this text segmentation task, the training goal of the machine learning model 105 to be trained is to make the segmentation result of the third predicted text description predicted by the machine learning model 105 as close as possible to the segmentation result of the third true text description. Therefore, in this pre-training phase, the corresponding training loss can be defined based on the difference between the segmentation result of the third predicted text description and the segmentation result of the third true text description. The training loss can be expressed as any appropriate loss function that can judge the difference between the segmentation result of the third predicted text description and the segmentation result of the third true text description. The difference between the segmentation result of the third predicted text description and the segmentation result of the third true text description can be minimized or reduced to meet the corresponding predetermined goal.

[0116] In some embodiments, in the first training sample, the information corresponding to each video segment may include identification information of the video segment. The identification information of the video segment may include, for example, an ID number of each video segment, or may include a resource path such as a webpage address of each video segment. It should be understood that the identification information of the video segment may be any appropriate information for identifying the video segment or being able to locate the position of the video segment.

[0117] Alternatively or additionally, in some embodiments, the information corresponding to each video segment may further include a start symbol indicating the beginning of the information corresponding to the video segment, and an end symbol indicating the end of the information corresponding to the video segment. The information corresponding to each video segment may be between the start symbol and the end symbol. The start symbol and the end symbol may be represented by appropriate characters, labels or other symbols, etc., which are not limited here. Through the start symbol and the end symbol of each video segment, each video segment can be quickly identified.

[0118] Alternatively or additionally, in some embodiments, the information corresponding to each video segment may also include at least one video frame extracted from the video segment and a rewriting result of the text description corresponding to the video segment. The rewriting result may be, for example, a rewriting result of the text description corresponding to the extracted video frame. The rewriting result may be used in the training phase of the machine learning model 105, which is conducive to improving the accuracy of the model prediction and enhancing the robustness of the model.

[0119] Alternatively or additionally, in some embodiments, the information corresponding to each video segment may further include a target number of characters that matches the video segment. The target number of characters may be a target number of characters for a text description of the audio corresponding to the video segment. By setting the target number of characters that matches the video segment, the duration of the video segment may be matched with the duration of the corresponding audio.

[0120] The following will discuss in detail the construction method of the training data set. The training data set of the model in the training phase and the test data set in the test phase can both be included in the data set of the model. It should be understood that the construction of the data set of the model can uniformly adopt the construction method discussed below. In other embodiments, the construction of the test data set of the model can also adopt other appropriate construction methods separately.

[0121] In some embodiments, the electronic device 110 may construct a training data set in the following manner. The electronic device 110 may divide the video material into a group of video clips to obtain multiple groups of video clips. Then, based on the selection requirements for the multiple groups of video clips, the electronic device 110 may select at least one group of video clips from the multiple groups of video clips, and construct the training data set based on the at least one group of video clips.

[0122] When constructing a training dataset, the video material may include a large amount of video material about multiple sample objects. Each video material may be split into a series of video clips. For example, in the field of video advertising, the video material may be an advertising video for multiple products.

[0123] In some embodiments, when segmenting the video material into a group of video segments, if the video material includes a text description, the electronic device 110 can use a speech recognition model to extract the corresponding first text description and the timestamps of multiple punctuation marks in the first text description from the video material. Then, based on the timestamps of the multiple punctuation marks, the electronic device 110 can segment the video material into a group of video segments. The speech recognition model can be, for example, an ASR (automatic speech recognition) model.

[0124] In some embodiments, if the video material does not have a text description, based on the degree of change of the visual features of each video frame in the video material, the electronic device 110 can segment the video material into a group of video segments. As an example, the visual feature difference between the continuous frames and the split frames of the video frames of the video material can be measured (the difference can be set to be greater than a preset threshold), so as to segment the video material into a group of video segments.

[0125] In some embodiments, the selection requirement may include a requirement for the number of video segments in each of the multiple groups of video segments. Alternatively or additionally, in some embodiments, the selection requirement may also include a requirement for the text fluency of the text description extracted from the video material. Alternatively or additionally, in some embodiments, the selection requirement may also include a requirement for the relevance between the text description and the object information.

[0126] Such selection requirements are intended to retain high-quality video materials, which can then be optimized and segmented through manual annotation or tool annotation. For example, screening criteria may include: the video length should not exceed 120 seconds or other duration; the number of video clips should be between 2 and 8, or within other set ranges; the text description should be fluent; the text description should be related to the video and object information. In this way, high-quality training data sets can be screened out.

[0127] Based on such a training dataset, a machine learning model is used to perform the video track remixing task, which can correctly select and sort the given unordered clips. A machine learning model is used to perform the script prediction task of the audio track, aiming to generate attractive and logically relevant spoken text for each video clip. A machine learning model can also be used to perform the script segmentation task of the subtitle track, including predicting appropriate punctuation for the spoken text. The output of the composite training task of all three tracks can be post-processed into JSON protocol or other suitable protocols. JSON protocol or other suitable protocols can be converted into video creatives, so that the quality can be systematically evaluated.

[0128] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 5 An exemplary structural block diagram of an apparatus 500 for model evaluation according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0129] like Figure 5 As shown, the device 500 includes an acquisition module 510, which is configured to acquire a test data set, the test data set includes a test sample, the test sample includes object information for a test object and multiple video clips; a generation module 520, which is configured to use a trained machine learning model to generate a predicted video clip sequence and its corresponding text description and a segmentation result of the text description based on the test sample, the video clip sequence includes at least two video clips arranged in sequence from multiple video clips; a first determination module 530, which is configured to determine the metric values ​​of performance indicators corresponding to each of the predicted video clip sequence, the text description and the segmentation result of the text description; and a second determination module 540, which is configured to determine the model performance of the trained machine learning model based on the metric values ​​of the performance indicator.

[0130] In some embodiments, the first determination module 530 is further configured to utilize another machine learning model to generate predicted metric values ​​of the performance indicators respectively corresponding to the video clip sequence, the text description, and the segmentation results of the text description based on the scoring requirements for the performance indicators respectively corresponding to at least one of the video clip sequence, the text description, and the segmentation results of the text description.

[0131] In some embodiments, the first determination module 530 is further configured to determine a measure value of the overlap performance index based on an intersection of video segments included between the predicted video segment sequence and the true video segment sequence corresponding to the test sample; and to determine a measure value of the accuracy performance index based on whether the predicted video segment sequence is identical to the video segments arranged in sequence in the true video segment sequence.

[0132] In some embodiments, the first determination module 530 is further configured to determine a measurement value of at least one of the following performance indicators corresponding to the predicted text description: authenticity indicator, text fluency indicator, contextual coherence indicator, logic indicator, correlation indicator between the text description and the predicted video frames in the video clip sequence, and correlation indicator between the text description and the object information.

[0133] In some embodiments, the first determination module 530 is further configured to determine a measure of a character length indicator of the text description based on a difference between a predicted target number of characters of the video segment sequence and a number of characters included in the text description.

[0134] In some embodiments, the first determination module 530 is further configured to determine a metric value of a performance indicator corresponding to the segmentation result based on whether the segmentation result of the text description meets at least one segmentation requirement, wherein the at least one segmentation requirement indicates at least one of the following: a requirement on the number of characters of the text segmentation unit corresponding to each video frame in the video clip sequence, a requirement on the number of text segmentation units divided by the text content between specified punctuation marks of the text description, and a requirement on the segmentation of some words in the text description.

[0135] The units and / or modules included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 700 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0136] It should be understood that one or more steps in the above method may be performed by an appropriate electronic device or a combination of electronic devices, such as the computing device 110 in FIG. 1 .

[0137] Figure 6 1 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The electronic device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown may be used to implement the electronic device 110 of FIG. 1 or Figure 5 Device 500.

[0138] like Figure 6 As shown, the electronic device 600 is in the form of a general electronic device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0139] The electronic device 600 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 600.

[0140] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0141] The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0142] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0143] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0144] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0145] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0146] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0147] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as an update, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0148] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for model evaluation, comprising: Acquire a test data set, wherein the test data set includes a test sample, and the test sample includes object information for a test object and a plurality of video clips; Using the trained machine learning model, based on the test sample, generating a predicted video clip sequence and its corresponding text description and a segmentation result of the text description, wherein the video clip sequence includes at least two video clips arranged in sequence among the multiple video clips; Determining metric values ​​of performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation result of the text description; as well as Based on the metric value of the performance indicator, determine the model performance of the trained machine learning model.

2. The method according to claim 1, wherein determining the metric values ​​of the performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation result of the text description comprises: Utilizing another machine learning model, based on the scoring requirements for the performance indicators respectively corresponding to the video clip sequence, the text description, and at least one of the segmentation results of the text description, a metric value of the predicted performance indicator respectively corresponding to the video clip sequence, the text description, and at least one of the segmentation results of the text description is generated.

3. The method according to claim 1, wherein determining a metric value of a performance indicator corresponding to the predicted video clip sequence comprises at least one of the following: Determining a metric value of a coincidence performance indicator based on an intersection of video segments included between the predicted video segment sequence and a true video segment sequence corresponding to the test sample; and A measure value of an accuracy performance indicator is determined based on whether the predicted video segment sequence is identical to each of the video segments arranged in sequence in the true video segment sequence.

4. The method according to claim 1, wherein determining a metric value of a performance indicator corresponding to the predicted text description comprises: Determine a metric value of at least one of the following performance indicators corresponding to the predicted text description: Authenticity indicators, Text Fluency Index, Contextual coherence index, Logical indicators, a correlation indicator between the text description and the predicted video frames in the video clip sequence, A correlation index between the text description and the object information.

5. The method according to claim 1, wherein determining the metric value of the performance indicator corresponding to the predicted text description further comprises: Based on the difference between the predicted target number of characters of the video segment sequence and the number of characters included in the text description, a measure value of a character length indicator of the text description is determined.

6. The method according to claim 1, wherein determining a metric value of a performance indicator corresponding to the predicted segmentation result of the text description comprises: determining a metric value of a performance indicator corresponding to the segmentation result based on whether the segmentation result of the text description meets at least one segmentation requirement, The at least one segmentation requirement indicates at least one of the following: The requirement on the number of characters in the text segmentation unit corresponding to each video frame in the video clip sequence, The requirement on the number of text segmentation units into which the text content between the specified punctuation marks of the text description is divided, The requirement for segmenting some words in the text description.

7. A device for model evaluation, comprising: An acquisition module is configured to acquire a test data set, wherein the test data set includes a test sample, and the test sample includes object information for a test object and a plurality of video clips; A generating module is configured to generate a predicted video clip sequence and its corresponding text description and a segmentation result of the text description based on the test sample by using the trained machine learning model, wherein the video clip sequence includes at least two video clips arranged in sequence among the plurality of video clips; A first determination module is configured to determine the metric values ​​of the performance indicators corresponding to the predicted video clip sequence, the text description, and the segmentation result of the text description; as well as The second determination module is configured to determine the model performance of the trained machine learning model based on the measurement value of the performance indicator.

8. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 6 when executed by the at least one processing unit.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images