Method, device and equipment for evaluating streaming video understanding model of intelligent agent and medium

Abstract

The embodiment of the invention provides an evaluation method for a streaming video understanding model of an agent. The evaluation method can be applied to the technical field of artificial intelligence. The evaluation method for the streaming video understanding model of the intelligent agent comprises the following steps: obtaining evaluation video data according to a received model evaluation benchmark; segmenting the evaluation video data to generate evaluation input data; generating evaluation output data of the streaming video understanding model through the evaluation input data; and obtaining model evaluation data of the streaming video understanding model according to the evaluation output data and the model evaluation benchmark. The embodiment of the invention further provides an evaluation device and equipment for the streaming video understanding model of the intelligent agent, a storage medium and a program product.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically to the field of image processing technology, and more specifically to an evaluation method, device, equipment, medium and product for a streaming video understanding model of an intelligent agent. Background Art

[0002] Artificial Intelligence (AI) is an important driving force for the new round of scientific and technological revolution and industrial transformation. It is a new key technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. As an important part of intelligent science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine (i.e., intelligent agent) that can respond in a similar way to human intelligence.

[0003] In order to improve the interactive experience between intelligent agents and humans, OpenAI has launched a chatbot model called ChatGPT (ChatGenerative Pre-trained Transformer) in recent years. It can generate answers based on the patterns and statistical laws seen in the pre-training stage, and can also interact with humans according to the context of the chat, achieve natural and fluent communication with humans, and provide intelligent question-answering and text generation capabilities for various scenarios. Its powerful natural language processing capabilities and multimodal transformation capabilities enable it to be used in multiple scenarios and fields. After that, subsequent natural language processing (NPL) models developed based on ChatGPT began to focus on the interactive capabilities of video understanding models. However, there is still a lack of evaluation methods for the interactive capabilities of such video understanding models. For example, there is no conventional quantitative evaluation method for the evaluation of streaming video understanding models. Usually, subjective judgments can only be made based on targeted tests by testers, and efficient evaluation of streaming video understanding models cannot be accurately achieved. Summary of the invention

[0004] In view of at least one of the above problems, embodiments of the present invention aim to provide a method, apparatus, device, medium and product for evaluating a streaming video understanding model of an intelligent agent that can accurately and efficiently evaluate the data processing capability of the streaming video understanding model.

[0005] One aspect of an embodiment of the present invention provides an evaluation method for a streaming video understanding model of an intelligent agent, which includes: obtaining evaluation video data according to a received model evaluation benchmark; performing segmentation on the evaluation video data to generate evaluation input data; generating evaluation output data of the streaming video understanding model through the evaluation input data; and obtaining model evaluation data of the streaming video understanding model according to the evaluation output data and the model evaluation benchmark; wherein the model evaluation benchmark is evaluation standard data generated according to the streaming video understanding task and a preset question and answer data set required for the evaluation of the streaming video understanding model, wherein the preset question and answer data set includes a matching data set between questions and standard answers corresponding to different streaming video understanding tasks.

[0006] According to an embodiment of the present invention, obtaining evaluation video data according to a received model evaluation benchmark includes: generating a model evaluation benchmark according to a preset evaluation comprehension task; and obtaining evaluation video data corresponding to the preset evaluation comprehension task.

[0007] According to one embodiment of the present invention, segmenting the evaluation video data to generate evaluation input data includes: obtaining preset task questions in the preset assessment comprehension task; and segmenting the evaluation video data according to the time nodes corresponding to the preset task questions to generate evaluation input data.

[0008] According to an embodiment of the present invention, in generating evaluation output data of a streaming video understanding model by evaluating input data, it includes: inputting the evaluation input data into the streaming video understanding model in a simulated video stream input form to generate evaluation output data.

[0009] According to one embodiment of the present invention, in obtaining model evaluation data of a streaming video understanding model based on evaluation output data and a model evaluation benchmark, it includes: obtaining evaluation task answers corresponding to preset task questions in the evaluation output data; and generating model evaluation data based on the evaluation task answers and the evaluation benchmark answers in the model evaluation benchmark.

[0010] According to one embodiment of the present invention, the evaluation method of the streaming video understanding model of an intelligent agent also includes: performing training on the streaming video understanding model based on model evaluation data and preset instruction following data, wherein the preset instruction following data includes active reasoning data corresponding to image and text interlacing instructions, noise instructions, and interruption instructions.

[0011] Another aspect of the embodiments of the present invention provides an evaluation device for a streaming video understanding model of an intelligent agent, which includes a data acquisition module, a data segmentation module, an evaluation output module and an evaluation acquisition module. The data acquisition module is used to acquire evaluation video data according to the received model evaluation benchmark; the data segmentation module is used to perform segmentation on the evaluation video data to generate evaluation input data; the evaluation output module is used to generate evaluation output data of the streaming video understanding model through the evaluation input data; and the evaluation acquisition module is used to acquire model evaluation data of the streaming video understanding model according to the evaluation output data and the model evaluation benchmark.

[0012] Another aspect of an embodiment of the present invention provides an electronic device, comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the evaluation method of the streaming video understanding model of the above-mentioned intelligent agent.

[0013] Another aspect of an embodiment of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned method for evaluating the streaming video understanding model of the intelligent agent.

[0014] Another aspect of an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the evaluation method of the streaming video understanding model of the above-mentioned intelligent agent.

[0015] The method for evaluating a streaming video understanding model of an intelligent agent provided by an embodiment of the present invention can at least partially solve the problem of lack of a streaming video understanding model evaluation method in the related art, and thus can achieve at least one of the following technical effects: Compared with the inability of the traditional technology to systematically and accurately evaluate the streaming video understanding model, the method of the embodiment of the present invention can establish an evaluation benchmark for the streaming video understanding model through streaming video and active reasoning, and implement a comprehensive evaluation of the six subtasks in the above two aspects of streaming video understanding and active reasoning according to the evaluation benchmark, thereby achieving a comprehensive and accurate evaluation of the streaming video understanding model, so that the streaming video understanding model can have a more universal evaluation standard. Furthermore, the interactive capability can be added to the streaming video understanding model at a low cost in combination with the fine-tuning effect corresponding to the provided instruction following data set, thereby significantly improving the active reasoning capability of the streaming video understanding model.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which: Figure 1 A diagram schematically shows an application scenario of an evaluation method, apparatus, device, medium, and program product for a streaming video understanding model of an intelligent agent according to an embodiment of the present invention; Figure 2 A flowchart of a method for evaluating a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown; Figure 3A A diagram schematically shows an application scenario of action prediction of an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention; Figure 3B A diagram schematically illustrates an application scenario of a dynamic state tracking task of an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention; Figure 3C A schematic diagram of an application scenario of a multi-round dependent reasoning task of an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is shown; Figure 3D A diagram schematically shows an application scenario of an active reminder task of an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention; Figure 3E A diagram schematically shows an application scenario of a noise recognition task of an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention; Figure 3F A diagram schematically showing an application scenario of a speaker recognition task of an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention; Figure 4 A flowchart schematically shows another application scenario of the method for evaluating a streaming video understanding model of an intelligent agent according to an embodiment of the present invention; Figure 5 A structural block diagram schematically shows a device for evaluating a streaming video understanding model of an intelligent agent according to an embodiment of the present invention; and Figure 6 A block diagram of an electronic device suitable for implementing an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0018] The above-mentioned drawings are part of the specification of the embodiments of the present invention, which illustrate exemplary embodiments of the present invention. The attached drawings and the description of the specification are used together to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description of the drawings and the following specific implementations are only exemplary and illustrative, and they cannot limit the scope of the present invention. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0020] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0021] The terms “first”, “second”, etc. used in the present invention do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0022] The directional terms used in the present invention, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0023] The words “include,” “including,” “have,” “contain,” etc. used in the present invention are open-ended terms, meaning including but not limited to.

[0024] The term "and / or" used in the present invention includes any or all combinations of the items mentioned.

[0025] Regarding the present invention, "plurality" includes "two" and "more than two"; regarding the present invention, "plurality of groups" includes "two groups" and "more than two groups".

[0026] The terms "substantially" and "approximately" used in the present invention are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0029] At present, large language models are developing rapidly, and more and more models are beginning to focus on video understanding, especially for real-time video stream understanding, such as GPT-4o. The GPT-4o model is trained based on a large amount of data from the Internet. It is better at processing text and audio, accepting any combination of text, audio and images as input content, and generating any combination of text, audio and images as output content. It supports 50 languages ​​and can respond to audio input in 232 milliseconds at the fastest, almost reaching the human response level. Therefore, the GPT-4o model has "real-time" interaction, emotional expression, stronger visual function, excellent multi-language performance, and a response speed that is almost the same as that of a real person, setting a new benchmark in reasoning and audio translation. However, for streaming video understanding, if the GPT-4o model is to achieve a level of understanding close to that of humans, it needs to further improve its multi-step reasoning ability, be able to perceive and predict future actions based on the current state, and have a certain degree of interaction ability, the ability to be interrupted and actively generate replies.

[0030] Therefore, for streaming video understanding, it is particularly important to evaluate the reasoning and interaction capabilities of the streaming video understanding model. The existing technology does not yet have an evaluation method specifically for the streaming video understanding model. Therefore, it is impossible to provide systematic, accurate and intuitive feedback on the streaming video understanding model's multi-step reasoning ability, ability to perceive and predict future actions, and interaction ability.

[0031] In view of at least one of the technical problems existing in the above-mentioned prior art, the embodiments of the present invention aim to provide an evaluation method, apparatus, device, medium and product for a streaming video understanding model of an intelligent agent that can accurately and efficiently evaluate the data processing capability of the streaming video understanding model.

[0032] One aspect of an embodiment of the present invention provides a method for evaluating a streaming video understanding model of an intelligent agent, which includes: obtaining evaluation video data based on a received model evaluation benchmark; performing segmentation on the evaluation video data to generate evaluation input data; generating evaluation output data of the streaming video understanding model through the evaluation input data; and obtaining model evaluation data of the streaming video understanding model based on the evaluation output data and the model evaluation benchmark.

[0033] Figure 1 An application scenario diagram schematically illustrates an evaluation method, apparatus, device, medium, and program product for a streaming video understanding model of an intelligent agent according to an embodiment of the present invention.

[0034] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0036] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0037] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the evaluation method of the stream video understanding model of the intelligent agent provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the evaluation device of the stream video understanding model of the intelligent agent provided in the embodiment of the present invention can generally be set in the server 105. The evaluation method of the stream video understanding model of the intelligent agent provided in the embodiment of the present invention can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the evaluation device of the stream video understanding model of the intelligent agent provided in the embodiment of the present invention can also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The evaluation method of the streaming video understanding model of the intelligent agent of the disclosed embodiment is described in detail.

[0041] Figure 2 A flowchart of a method for evaluating a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0042] like Figure 2 As shown, one aspect of an embodiment of the present invention provides a method for evaluating a streaming video understanding model of an intelligent agent, which includes operations S201 to S204.

[0043] In operation S201, evaluation video data is acquired according to the received model evaluation benchmark; In operation S202, segmenting is performed on the evaluation video data to generate evaluation input data; and In operation S203, evaluation output data of the streaming video understanding model is generated by evaluating the input data.

[0044] In operation S204, model evaluation data of the streaming video understanding model is obtained according to the evaluation output data and the model evaluation benchmark; The model evaluation benchmark is evaluation standard data generated based on the streaming video understanding task and preset question and answer data set required for the evaluation of the streaming video understanding model, wherein the preset question and answer data set includes a matching data set between questions and standard answers corresponding to different streaming video understanding tasks.

[0045] The intelligent agent can be the executor of the evaluation method of the streaming video understanding model of the above-mentioned intelligent agent in the embodiment of the present invention, or it can be an executor controlled by the evaluation method of the streaming video understanding model. Specifically, it can be a humanoid intelligent robot or other AI device, which usually has an actuator to complete specific action tasks. For example, a humanoid robot can use a mechanical manipulator to complete the action task of picking up an object. Among them, the intelligent agent in the embodiment of the present invention needs to have a streaming video understanding model and provide a strong streaming video understanding interaction capability based on the streaming video understanding model.

[0046] The streaming video understanding model of the embodiment of the present invention can be understood as a multi-modal large model for video content understanding interaction of long-term video input streams. The streaming video understanding model can integrate multiple modal data such as text, images, videos, and audio, and perform comprehensive understanding and reasoning, with stronger understanding capabilities and wider application scenarios.

[0047] The model evaluation benchmark is the standard evaluation standard data generated according to the evaluation requirements of the streaming video understanding model, and can be specifically generated according to the streaming video understanding task required to be evaluated and the preset question-answering dataset. The preset question-answering dataset includes the matching datasets between questions and standard answers corresponding to different streaming video understanding tasks.

[0048] The evaluation video data may be streaming video set data having the question set data of the above-mentioned preset question and answer data set, and may have large-scale video streaming data, wherein the video streaming data may contain corresponding question set data.

[0049] The evaluation video data can be disassembled and extracted according to the time nodes in the video stream and the various task problems in the problem set data, and the problem set data corresponding to the specific stream video understanding tasks can be extracted. Among them, the evaluation input data is the segmented data of the evaluation video data, which belongs to the video stream data corresponding to certain time periods in the evaluation video data, and these video stream data can include at least one problem set of at least one stream video understanding task.

[0050] The evaluation input data is input into the streaming video understanding model, and the evaluation output data is output through the reasoning and interaction capabilities of the streaming video understanding model. The evaluation output data is the reasoning data of the streaming video understanding model, including the reasoning answer set corresponding to the question set in the evaluation input data.

[0051] The model evaluation data is obtained according to the matching relationship between the standard answer set in the preset question-answer data set that matches the streaming video understanding task and the corresponding inference answer set in the above evaluation output data that can be provided by the model evaluation benchmark. The model evaluation data is the evaluation data of the streaming video understanding ability generated according to the matching relationship between the inference answers in the above inference answer set and the corresponding standard answers in the standard answer set, such as scoring data.

[0052] Therefore, compared with traditional technologies that are unable to conduct a systematic and accurate evaluation of the streaming video understanding model, the method of the embodiment of the present invention can establish an evaluation benchmark for the streaming video understanding model, and implement a comprehensive evaluation of the above-mentioned streaming video understanding task based on the evaluation benchmark, thereby enabling a comprehensive and accurate evaluation of the streaming video understanding model, so that the streaming video understanding model can have a more universal evaluation standard.

[0053] like Figure 2-Figure 4 As shown, according to an embodiment of the present invention, in operation S201, obtaining evaluation video data according to the received model evaluation benchmark includes: Generate model evaluation benchmarks based on preset evaluation comprehension tasks; Obtain evaluation video data corresponding to the preset evaluation comprehension task.

[0054] The preset evaluation and understanding tasks can be the understanding tasks that need to be evaluated and preset by the user for the reasoning and evaluation scenarios of the streaming video understanding model. Specifically, the preset evaluation and understanding tasks of the embodiment of the present invention can be divided into two aspects: streaming video understanding and active reasoning. At the same time, streaming video understanding can be divided into three subtasks: action prediction task, dynamic state tracking task and multi-turn dependency reasoning task. Active reasoning can also be divided into three subtasks: proactive alerting task, proactive turn-taking task and speaker identification task.

[0055] Among them, each subtask of the preset evaluation understanding task can be used as the aforementioned streaming video understanding task, which can achieve understanding and reasoning for different question-answering interaction scenarios.

[0056] Figure 3A An application scenario diagram of action prediction of an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0057] Action prediction tasks can be used to determine the subsequent actions that the protagonist will take for a certain purpose based on the first-person perspective. Figure 3A As shown in FIG. 5 , based on the recognition of continuous frame video images, after receiving the question “I am tasked with Peel, chop and cook the second potato. What is next step?”, the streaming video understanding model can make an action guess of “pick up potato” based on the understanding of the streaming video.

[0058] Figure 3B An application scenario diagram of a dynamic state tracking task of an evaluation method for a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0059] The dynamic state tracking task can be used to understand the possible changes in the current state of the video as the video stream progresses. The video stream understanding model can combine the understanding of the video stream to perceive the possible changes that may occur in the current state of the video and give a reasonable response. Figure 3B As shown in the figure, the answers to the same question are different at different times. The model needs to answer these questions correctly. For example, for “How many cars have shown up?”, the streaming video understanding model needs to combine the understanding of the video images to give the corresponding number of cars at the time “00:05” as “1”, the corresponding number of cars at the time “00:10” as “2”, and the corresponding number of cars at the time “00:17” as “3”.

[0060] Figure 3C An application scenario diagram of a multi-round dependent reasoning task of the evaluation method of the streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0061] The multi-round dependency reasoning task is used to target the common front-to-back dependency relationships in streaming videos. Different questions are set at different moments, and the current question comes from the answer to the previous question. Figure 3C As shown, the continuous question-answering scenario: Q: Who is organizing the back pack? A:A woman Q: Who is talk to? A:A man Q: What is doing? A: Sitting on the bed. Figure 3D An application scenario diagram of an active reminder task of the evaluation method of the streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0062] The active reminder task is used to describe an event given to the streaming video understanding model at the beginning, hoping that the model can remind the user when the video progresses to this event. Figure 3D As shown, if “Inform me when acat is getting a shot” is provided, the streaming video understanding model will provide an “Informed” reminder at the moment “00:25” based on the understanding of the streaming video image.

[0063] Figure 3E An application scenario diagram of a noise recognition task of an evaluation method for a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0064] The noise recognition task can identify user commands that require a response and noise that does not require a response. Figure 3E As shown in the figure, for the question “What happens to the man in black clothes after he stands on the cliff?”, the streaming video understanding model can judge that it is a non-noise question and give the answer “He is caught by anet laid by a helicopter”. However, for the question “Oh well, it is what it is No point in overthinking it” which the streaming video understanding model judges to be a noisy question, the streaming video understanding model can remain silent ( <silence>) will not respond.

[0065] Figure 3F An application scenario diagram of a speaker recognition task of an evaluation method for a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0066] Speaker identification tasks can be used in multi-person conversation scenarios. The streaming video understanding model needs to figure out which speaker issued the current command in the streaming video. Usually, these characters can introduce themselves or others. Figure 3F As shown in the figure, based on the introductions of others "This is Bobi" and "This is Wanita", the streaming video understanding model can give the answer "Wanita (wants to served drink)" to the question "Whowants to serve drink?"

[0067] Therefore, different preset evaluation and understanding tasks can correspond to different reasoning and interaction capabilities of the streaming video understanding model. Considering that the evaluation video data can be a streaming video set data with the question set data of the above-mentioned preset question-answering data set, it can have large-scale video streaming data, wherein these video streaming data can contain corresponding question set data. These question set data can be used to match the questions of the video images in the video stream according to the above-mentioned preset evaluation and understanding tasks. Figure 3A The problem corresponding to the action prediction task provided is "I am tasked with Peel, chop and cook the second potato· What is next step".

[0068] The obtained evaluation video data may have 1121 videos, corresponding to 2290 question-answering datasets for 6 different streaming video understanding tasks. Therefore, the evaluation video data obtained according to the preset evaluation understanding tasks can have a wider task coverage, making the evaluation of the streaming video understanding model more targeted and convincing.

[0069] like Figure 2-Figure 4 As shown, according to an embodiment of the present invention, in operation S202, segmenting the evaluation video data to generate evaluation input data includes: Get the preset task questions in the preset assessment comprehension task; The evaluation video data is segmented according to the time nodes corresponding to the preset task problems to generate evaluation input data.

[0070] The evaluation video data can be an offline video. According to different preset evaluation understanding tasks (such as the streaming video understanding task of action prediction), the preset task problem matching the task can be determined, such as Figure 3A The problem corresponding to the action prediction task provided is "I am tasked with Peel, chop and cook the second potato· What isnext step".

[0071] Furthermore, the evaluation video data is segmented according to the time nodes that match the preset task questions in the video stream, and only the video stream data in the time period directly related to the preset task questions can be obtained as the evaluation input data, which can be understood as intercepting the images of the video stream according to the time nodes corresponding to the preset task questions. The preset task questions are the preset questions in the above evaluation video data that match the stream video understanding task, and usually have a known matching standard answer. Figure 3B As shown in the figure, in the video stream corresponding to the preset task question "How many cars have shown up?" of the dynamic state tracking task, cars appear in the images at "00:05", "00:10" and "00:17" respectively. Then, the video between at least "00:05" and "00:17" can be used as the evaluation input data, or the streaming video images at three times, "00:05", "00:10" and "00:17", can be used as the evaluation input data, thereby completing the segmentation of the evaluation video data.

[0072] Therefore, even if most video understanding models cannot support streaming video input, the offline video can be segmented according to the time nodes corresponding to different tasks, and the segmented input can be input into the streaming video understanding model to be evaluated to complete the simulated input effect of the video stream. In this way, invalid video understanding evaluation can be greatly avoided, and evaluation efficiency and accuracy can be improved.

[0073] like Figure 2-Figure 4 As shown, according to an embodiment of the present invention, in operation S203, by evaluating the input data, generating the evaluation output data of the streaming video understanding model includes: The simulated video stream input form inputs the evaluation input data into the streaming video understanding model to generate the evaluation output data.

[0074] Since the streaming video understanding model needs to implement reasoning and understanding interaction on the continuous input video stream during the application of the intelligent agent, the simulation of the video input can be realized through the segmentation and acquisition of the above-mentioned evaluation input data, so that the evaluation input data can be input into the streaming video understanding model in the form of a video stream in the order of time nodes. Among them, the video images at different time nodes in the video stream of the evaluation input data can correspond to different preset task problems.

[0075] The streaming video understanding model processes the video stream images of the input evaluation input data to generate reasoning answers to the preset task questions corresponding to the preset evaluation understanding tasks in the evaluation input data, and these reasoning answers can constitute the processing result data set of the streaming video understanding model. Among them, the evaluation output data is the reasoning data of the streaming video understanding model, including the reasoning answer set corresponding to the question set in the evaluation input data.

[0076] Therefore, by simulating the input form of the video stream and inputting the evaluation input data into the streaming video understanding model, the processing process of the streaming video understanding model can be made more in line with the actual application scenario, the output evaluation output data can be more credible, and the efficiency and accuracy of the model processing can be guaranteed at the same time.

[0077] like Figure 2-Figure 4 As shown, according to one embodiment of the present invention, in operation S204, obtaining model evaluation data of the streaming video understanding model according to the evaluation output data and the model evaluation benchmark includes: Obtain the evaluation task answers corresponding to the preset task questions in the evaluation output data; Generate model evaluation data based on the answers to the evaluation tasks and the evaluation benchmark answers in the model evaluation benchmark.

[0078] For the preset task questions corresponding to different streaming video understanding tasks in the evaluation input data, a set of inference answers corresponding to the streaming video understanding tasks can be generated after processing by the streaming video understanding model, that is, the above preset task questions can all be answered by the model.

[0079] The evaluation task answers are the reasoning answers to the different preset task questions corresponding to the different stream video understanding tasks, and the evaluation benchmark answers are the preset standard answers to the different preset task questions corresponding to the different stream video understanding tasks. The evaluation benchmark answers can be obtained from the matching dataset of the question-standard answer of the preset question-answer dataset of the model evaluation benchmark.

[0080] By comparing the answer to the evaluation task with the answer to the evaluation benchmark in the model evaluation benchmark, the accuracy of the answer to the evaluation task corresponding to the streaming video understanding task can be calculated. Furthermore, the accuracy of each evaluation task answer of all 6 different streaming video understanding tasks can be calculated, and the average of these accuracy rates can be taken as a basis for judging the strength of the streaming video understanding and reasoning capabilities of the streaming video understanding model. Among them, the model evaluation data can be the evaluation result of the streaming video understanding and reasoning capabilities of the above streaming video understanding model.

[0081] For example, for Figure 3A For the action prediction task shown in the figure, if the streaming video understanding model gives the evaluation task answer "pick up potato", then the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task is 100%. Correspondingly, if the streaming video understanding model gives the evaluation task answer "pick up tomato", then the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task can be 60%.

[0082] like Figure 3B For the dynamic state tracking task shown in the figure, if the evaluation task answer given by the streaming video understanding model is: the number of cars corresponding to the time "00:05" is "1", the number of cars corresponding to the time "00:10" is "2", and the number of cars corresponding to the time "00:17" is "3", then the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task is 100%. However, if the evaluation task answer given by the streaming video understanding model is: the number of cars corresponding to the time "00:05" is "1", the number of cars corresponding to the time "00:10" is "2", and the number of cars corresponding to the time "00:17" is "2", then the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task can be 66%.

[0083] like Figure 3C For the multi-round dependency reasoning task shown in the figure, if the answer to the evaluation task given by the streaming video understanding model does not involve the previous and subsequent dependencies of "the current question comes from the answer to the previous question", then even if the streaming video understanding model gives reasonable answers to questions that appear at different times, the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task may be only 20%.

[0084] like Figure 3D For the active reminder task shown in the figure, if the evaluation task answer given by the streaming video understanding model does not provide the user with the reminder "Informed" at the moment "00:25" of "a cat is getting a shot", then the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task can be 0%. If the evaluation task answer given by the streaming video understanding model provides the reminder "Informed" at the moment "00:24" one second before the moment "00:25" of "a cat is getting a shot", then the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task can be 90%.

[0085] like Figure 3E For the noise recognition task shown in the figure, if the evaluation task answer given by the streaming video understanding model provides the user with a non- <silence>If the response is , the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task can be 0%, and the corresponding accuracy value can be matched depending on the degree of the response.

[0086] like Figure 3F For the speaker recognition task shown in the figure, if the answer to the evaluation task given by the streaming video understanding model is not "Wanita" but "Bob", then the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task can be 50%. If the answer to the evaluation task given by the streaming video understanding model is neither "Wanita" nor "Bob", then the reasoning and understanding accuracy of the streaming video understanding model for the action prediction task can be 0%.

[0087] Finally, the reasoning and understanding accuracy of the above six different streaming video understanding tasks are averaged, which can be used as the overall reasoning and understanding accuracy of the streaming video understanding task, constituting the final model evaluation data of the streaming video understanding task.

[0088] In summary, compared with the inability of traditional technologies to systematically and accurately evaluate streaming video understanding models, the method of the embodiment of the present invention can establish an evaluation benchmark for streaming video understanding models through streaming video and active reasoning, which is the first evaluation standard in this field. Moreover, according to the evaluation benchmark, a comprehensive evaluation of the six subtasks in the above-mentioned streaming video understanding and active reasoning can be achieved, thereby achieving a comprehensive and accurate evaluation of the streaming video understanding model, so that the streaming video understanding model can have a more universal evaluation standard.

[0089] like Figure 2-Figure 4 As shown, according to one embodiment of the present invention, the evaluation method of the stream video understanding model of the intelligent agent also includes: The streaming video understanding model is trained based on the model evaluation data and the preset instruction following data, wherein the preset instruction following data includes active inference data corresponding to the image and text interleaving instructions, the noise instructions, and the interruption instructions.

[0090] Based on the above model evaluation data, the understanding and reasoning interaction capabilities of the streaming video understanding model can be intuitively judged. Therefore, when the interaction capabilities of the streaming video understanding model are poor, further training of the streaming video understanding model can be considered to improve the overall understanding and reasoning interaction capabilities of the model. On the other hand, considering that the current streaming video understanding model, as a multimodal large model, has developed rapidly, it is seriously lacking in active reasoning and interaction capabilities, such as the ability to recognize interruptions and noise, which means that its human-computer interaction experience still has a high room for improvement. It is also possible to consider further training the streaming video understanding model with strong reasoning and understanding interaction capabilities for the above six tasks to improve its human-computer interaction experience.

[0091] In view of the above situation, the preset instruction following data of the embodiment of the present invention can be a data set with a small amount of data in a multimodal instruction following format for images, texts and interlaced images and texts. By fine-tuning the data of the preset instruction following data, the interactive capability can be added to the streaming video understanding model of the multimodal large model at a lower cost. Among them, the data set of the preset instruction following data mainly includes the following three parts: interlaced image and text instructions, noise instructions and interruption instructions. Among them, the interlaced image and text instructions refer to the training pictures and instructions that will appear in the training data interlaced to improve the active reasoning ability of the model; secondly, the noise instructions can be used as negative samples to enhance the robustness of the model, such as "I understand, this video shows a flag flying"; in addition, the interruption instruction adds the interruption capability to the model, such as "Sorry, interrupt for a moment". Therefore, by training the above-mentioned streaming video understanding model that obtains the above-mentioned model evaluation data based on the data set of the preset instruction following data, the streaming video understanding model that has completed the evaluation can be injected with the ability of active reasoning and interaction, thereby significantly improving its level of streaming video understanding and reasoning.

[0092] Figure 4 A flowchart of another application scenario of the evaluation method of the streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0093] like Figure 4 As shown in the figure, because of the use of interleaved text and image instructions, the active reasoning of the streaming video understanding model that completes the preset instruction following data training can be regarded as the future time positioning. This method can be applied to the KV-cache storage video stream (Video Streaming) of the Omni Language Model (OmniLanguage Model). Combined with the maximum heap algorithm (HighlightSpot Max-Heap), when the video clip of a certain event node continues to maintain a high attention score, the model can issue an active reminder at this point. For the interactive ability of the large model, because it has been trained on noise instructions and interruption (stop) instructions, the factual interactive dialogue including proactive interruption can be realized through parallel decoding (Parallel Decoding).

[0094] Therefore, based on the above evaluation of the reasoning and understanding interaction capabilities of the streaming video understanding model, we can further follow the fine-tuning effect corresponding to the data set according to the preset instructions provided, and add interactive capabilities to the streaming video understanding model at a lower cost, thereby significantly improving the active reasoning capabilities of the streaming video understanding model.

[0095] In order to add active reasoning capabilities to the current multimodal model, the above method of the embodiment of the present invention provides an instruction following data set, and fine-tuning on this data set can realize the active reasoning capabilities of the model. Furthermore, through the evaluation of the streaming video understanding model and injecting interactive functions into the current multimodal large language model, the streaming video understanding model can realize the ability of intelligent "interruption and noise recognition".

[0096] Compared with the inability of the traditional technology to systematically and accurately evaluate the streaming video understanding model, the method of the embodiment of the present invention can establish an evaluation benchmark for the streaming video understanding model through streaming video and active reasoning, and implement a comprehensive evaluation of the six subtasks in the above two aspects of streaming video understanding and active reasoning according to the evaluation benchmark, thereby achieving a comprehensive and accurate evaluation of the streaming video understanding model, so that the streaming video understanding model can have a more universal evaluation standard. Furthermore, the interactive capability can be added to the streaming video understanding model at a low cost in combination with the fine-tuning effect corresponding to the provided instruction following data set, thereby significantly improving the active reasoning capability of the streaming video understanding model.

[0097] Based on the above-mentioned evaluation method of the stream video understanding model of the intelligent agent, the present invention also provides an evaluation device for the stream video understanding model of the intelligent agent. Figure 5 The device is described in detail.

[0098] Figure 5 The structure block diagram of the evaluation device of the streaming video understanding model of the intelligent agent according to an embodiment of the present invention is schematically shown.

[0099] like Figure 5 As shown, the evaluation device 500 of the stream video understanding model of the intelligent agent in this embodiment includes a data acquisition module 510, a data segmentation module 520, an evaluation output module 530 and an evaluation acquisition module 540.

[0100] The data acquisition module 510 is used to acquire the evaluation video data according to the received model evaluation benchmark. In one embodiment, the data acquisition module 510 can be used to perform the operation S201 described above, which will not be described in detail here.

[0101] The data segmentation module 520 is used to segment the evaluation video data to generate evaluation input data. In one embodiment, the data segmentation module 520 can be used to perform the operation S202 described above, which will not be described in detail here.

[0102] The evaluation output module 530 is used to generate evaluation output data of the streaming video understanding model by evaluating the input data. In one embodiment, the evaluation output module 530 can be used to perform the operation S203 described above, which will not be described in detail here.

[0103] The evaluation acquisition module 540 is used to acquire model evaluation data of the streaming video understanding model according to the evaluation output data and the model evaluation benchmark. In one embodiment, the evaluation acquisition module 540 can be used to perform the operation S204 described above, which will not be described in detail here.

[0104] According to an embodiment of the present invention, any multiple modules among the data acquisition module 510, the data segmentation module 520, the evaluation output module 530 and the evaluation acquisition module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the data acquisition module 510, the data segmentation module 520, the evaluation output module 530 and the evaluation acquisition module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the data acquisition module 510, the data segmentation module 520, the evaluation output module 530 and the assessment acquisition module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0105] Figure 6 A block diagram of an electronic device suitable for implementing an evaluation method of a streaming video understanding model of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0106] The electronic device provided by an embodiment of the present invention includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the evaluation method of the streaming video understanding model of the above-mentioned intelligent agent.

[0107] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0108] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0109] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0110] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned method for evaluating the streaming video understanding model of the intelligent agent.

[0111] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiment; or it may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0112] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0113] An embodiment of the present invention also includes a computer program product, which includes a computer program, which, when executed by a processor, implements the evaluation method of the streaming video understanding model of the above-mentioned intelligent agent.

[0114] The computer program includes program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0115] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0116] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0117] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0118] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed completely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0121] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.< / silence> < / silence>

Claims

1. A method for evaluating a streaming video understanding model of an intelligent agent, characterized in that: include: Obtaining evaluation video data according to the received model evaluation benchmark; Segmenting the evaluation video data to generate evaluation input data; Generating evaluation output data of the streaming video understanding model through the evaluation input data; as well as Acquire model evaluation data of the streaming video understanding model according to the evaluation output data and the model evaluation benchmark; The model evaluation benchmark is evaluation standard data generated based on the streaming video understanding task and preset question and answer data set required for the evaluation of the streaming video understanding model, wherein the preset question and answer data set includes a matching data set between questions and standard answers corresponding to different streaming video understanding tasks.

2. The method according to claim 1, characterized in that The step of obtaining the evaluation video data according to the received model evaluation benchmark includes: Generating the model evaluation benchmark according to a preset evaluation comprehension task; Acquire evaluation video data corresponding to the preset evaluation comprehension task.

3. The method according to claim 2, characterized in that The step of segmenting the evaluation video data to generate evaluation input data includes: Obtaining a preset task question in the preset assessment comprehension task; The evaluation video data is segmented according to the time nodes corresponding to the preset task problems to generate evaluation input data.

4. The method according to claim 1, characterized in that: In the step of generating the evaluation output data of the streaming video understanding model through the evaluation input data, the step includes: The evaluation input data is input into the streaming video understanding model in a simulated video stream input form to generate the evaluation output data.

5. The method according to claim 3, characterized in that: In the obtaining of the model evaluation data of the streaming video understanding model according to the evaluation output data and the model evaluation benchmark, including: Obtaining the evaluation task answer corresponding to the preset task question in the evaluation output data; The model evaluation data is generated according to the evaluation task answer and the evaluation benchmark answer in the model evaluation benchmark.

6. The method according to claim 1, characterized in that Also includes: The streaming video understanding model is trained according to the model evaluation data and preset instruction following data, wherein the preset instruction following data includes active reasoning data corresponding to image and text interlacing instructions, noise instructions, and interruption instructions.

7. An evaluation device for an intelligent agent's streaming video understanding model, characterized in that: include: A data acquisition module, used to acquire evaluation video data according to the received model evaluation benchmark; A data segmentation module, used for segmenting the evaluation video data to generate evaluation input data; An evaluation output module, used to generate evaluation output data of the streaming video understanding model through the evaluation input data; as well as An evaluation acquisition module is used to obtain model evaluation data of the streaming video understanding model based on the evaluation output data and the model evaluation benchmark.

8. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images