Unified platform and reference-based multi-modal large model question and answer evaluation system

Abstract

The invention provides a multi-modal large model question and answer evaluation system based on a unified platform and benchmark. The system comprises a visual question and answer evaluation platform, a question and answer benchmark based on a unified capability framework and an evaluation strategy based on a large language model. The visual question and answer evaluation platform is a unified platform which is based on concurrency control and supports multi-modal data; the systematic evaluation of the model performance is realized through a standardized data structure and task design according to the homemade question and answer benchmark; the evaluation strategy scores the answers based on multiple independent scores and a semantic comparison mechanism. According to the method, efficient task scheduling and stable performance expression are realized, and the requirements of a high-load evaluation scene can be met; systematic evaluation of different model performances is realized, and all core capabilities required by questions and answers of owners are covered. And the answers are scored based on multiple independent scoring and a semantic comparison mechanism, so that the scoring accuracy and consistency are improved, and the limitation of a traditional evaluation method in processing complex answers is overcome.

Description

Technical Field

[0001] The present invention relates to the field of embodied question-answering technology, and in particular to a multimodal large-model embodied question-answering evaluation system based on a unified platform and benchmark. Background Art

[0002] Embodied Question Answering (EQA) is a multimodal reasoning task that requires the model to understand and answer questions based on visual and language information in an embodied environment. EQA tasks typically involve visual information V and a natural language question L posed by the user. The answering process can be described as: A = f EQA (V,L), where f EQA The reasoning process by which the model combines visual information with the verbal question to generate the answer A naturally aligns with the reasoning process of multimodal large models (MLMs). The difficulty of this task lies in the fact that the model must not only understand the content in the visual environment but also make reasonable inferences based on the verbal question. Embodied question answering tasks include open-ended questions and multiple-choice questions. In recent years, a number of embodied question answering benchmarks have been proposed, evaluating models' object recognition, affordance, spatiotemporal understanding, and task planning capabilities using simulators, scanned scenes, or real-world visual data. However, these benchmarks suffer from problems such as a single capability dimension, insufficient scenario diversity, and limited data volume. Furthermore, the data formats of different benchmarks are not standardized, and some data relies on other datasets, making the download and extraction process cumbersome and increasing the complexity of cross-benchmark evaluation.

[0003] Evaluation Metric. Traditional evaluation of open-ended questions usually uses metrics such as BLEU, ROUGE, and CIDEr to measure the similarity between the generated answer and the reference answer. In the multiple-choice question scenario, the accuracy metric is often used for evaluation. Although these traditional metrics are intuitive, they have limitations in complex embodied question-answering tasks and it is difficult to fully measure the reasoning ability of the model. In recent years, large language models (LLMs). Large models for short have been introduced into the evaluation process. By using questions, answers, and scoring rules as text prompts, large language models can use their common sense reasoning ability to score the generated answers. In this way, the evaluation of LLMs can be made more comprehensive.

[0004] However, current large-scale model evaluation platforms still have several limitations. These include: a single evaluation method, excessive reliance on subjective evaluation, and a lack of multi-benchmark reasoning capabilities, making it difficult to fully reflect the model's performance across different tasks and failing to cover all the core capabilities required for embodied question answering. Furthermore, they lack multimodal support, are unable to handle complex multi-image or video inputs, and are unable to adapt to the demands of high-load evaluation scenarios, thus limiting the evaluation of embodied question answering. Furthermore, the evaluation platforms are relatively limited in functionality and lack support for the entire evaluation process, resulting in a complex and inefficient process. Summary of the Invention

[0005] The technical problem to be solved by the present invention is the need to provide a multimodal large-model embodied question-answering evaluation system based on a unified platform and benchmark, aiming to adopt concurrent control and distributed architecture to achieve efficient task scheduling and stable performance, and adapt to the needs of high-load evaluation scenarios; on this basis, it also covers all the core capabilities required for embodied question-answering based on the embodied question-answering benchmark of the unified capability framework, realizes the systematic evaluation of the performance of different models, and comprehensively reflects the performance of the model in different tasks; and further scores the answers based on multiple independent scoring and semantic comparison mechanisms, optimizes the evaluation strategy, improves the accuracy and consistency of the scoring, overcomes the limitations of traditional evaluation methods in dealing with complex answers, and effectively promotes the organic integration and development of multimodal intelligence and embodied question-answering fields.

[0006] To this end, the present invention provides a multimodal large-model embodied question-answering evaluation system based on a unified platform and benchmark, comprising a visual embodied question-answering evaluation platform, an embodied question-answering benchmark based on a unified capability framework, and an evaluation strategy based on a large language model. The visual embodied question-answering evaluation platform is a unified platform based on concurrent control and supporting multimodal data. The embodied question-answering benchmark based on a unified capability framework implements a systematic evaluation of model performance through standardized data structures and task design. The evaluation strategy based on a large language model scores answers based on multiple independent scoring and semantic comparison mechanisms.

[0007] The implementation process of the visual embodied question-answering evaluation platform includes the following steps:

[0008] Step S101: Using the Gradio framework, the front end of the visual embodied question-answering evaluation platform is constructed, comprising a direct dialogue module, a model evaluation module, and a result analysis module. The direct dialogue module is used to conduct multiple rounds of interaction with the multimodal large model to implement multimodal data input and real-time feedback. The model evaluation module selects different multimodal large models and benchmark tasks for combined execution to achieve performance evaluation, and quantitatively analyzes the results using preset evaluation indicators. The result analysis module displays the scores and rankings of each multimodal large model through at least one of a table, a bar chart, and a radar chart.

[0009] Step S102: Implement the backend of the visual embodied question-answering evaluation platform based on the FastAPI architecture and the uvicorn server. The backend adopts a distributed architecture including a model worker, a controller, and a server. The model worker is responsible for loading and running different multimodal large models and reporting node status to the controller through a heartbeat mechanism. The controller is used to dynamically allocate inference requests based on the load of the model worker and perform load transfer and node recovery when a node fails. The server is responsible for receiving front-end requests and returning inference results, thereby achieving docking between the front-end and back-end.

[0010] Step S103: Use the LOCUST tool to perform a stress test on the visual embodied question-answering evaluation platform;

[0011] The implementation process of the embodied question-answering benchmark based on the unified capability framework includes the following sub-steps:

[0012] Step S201: Building a unified capability framework based on the five capability dimensions of embodied intelligence: object understanding, knowledge, spatiotemporal perception, task understanding, and planning.

[0013] In step S202, different data sets are integrated and screened, the data in the data sets are classified based on the capability framework of step S201, and the data format is unified to construct an embodied question-answering benchmark.

[0014] A further improvement of the present invention is that the visual embodied question-and-answer evaluation platform is based on a distributed architecture and supports multiple different multimodal large models to be deployed on different nodes at the same time; the node status supervision method includes passive detection method and active detection method; under the passive detection method, the working node regularly sends heartbeat messages to the registration center. If no heartbeat message is received within a preset number of consecutive cycles, the registration center will determine that the node has crashed and automatically remove it from the service list; under the active detection method, the registration center regularly sends detection messages to all nodes. If no response is received from the node within a preset number of consecutive cycles, the node will be actively removed.

[0015] A further improvement of the present invention is that in step S101, the front end uses an interactive front end tool to submit various types of input data, view evaluation feedback in real time, and simultaneously collaborate with the back end through a unified interface.

[0016] A further improvement of the present invention is that in step S102, a unified interface is first defined in a distributed architecture through the FastAPI architecture so that tasks can be distributed and processed between different nodes; then, through the concurrent connection of the uvicorn server, combined with the asynchronous streaming processing model of the FastAPI architecture, data transmission and processing in high-concurrency scenarios are realized; the asynchronous streaming processing model of the FastAPI architecture is used to split the data processing process into non-blocking tasks to realize the processing of multimodal data, wherein the multimodal data includes images, text and video.

[0017] A further improvement of the present invention is that in step S102, an asynchronous queue mechanism is used to implement data transmission and processing in a high-concurrency scenario. When multiple users initiate requests simultaneously, tasks are dynamically allocated according to the shortest queue principle. The asynchronous queue mechanism is used to implement the asynchronous queue request processing flow of the model worker, including the following sub-steps:

[0018] Step S1021, global initialization, initialization batch size batch_size = 20, maximum waiting time batch_timeout = 1 second, request queue request_queue is empty;

[0019] Step S1022: Use await to wait for the asynchronous request to complete, obtain the keyword of the asynchronous request, and extract the request parameter params from the asynchronous request;

[0020] Step S1023, creating an object future through the asyncio.Future() function to store the result of the request processing;

[0021] Step S1024, putting the request parameters params and the object future into the request queue request_queue;

[0022] Step S1025, determine whether the length of the request queue request_queue is greater than the set batch size batch_size. If so, create an asynchronous task through the asyncio.create_task() function to process the current batch request; if not, jump to step S1026;

[0023] Step S1026, use the await asyncio.wait_for() function to wait for the maximum timeout period batch_timeout. Within the timeout period, determine whether the request queue request_queue is empty. If it is not empty, create an asynchronous task through the asyncio.create_task() function to process the current batch request, and jump to step S1027; if it is empty, jump directly to step S1027;

[0024] Step S1027: Use await future to obtain the inference result, and return the inference result encapsulated in JSON format based on the JSONResponse tool.

[0025] A further improvement of the present invention is to create an asynchronous task through the asyncio.create_task() function to process the current batch of requests, and its implementation process includes the following steps:

[0026] Step A1: Determine whether the request queue request_queue is empty. If so, directly return an empty result or an exception reminder that the queue is empty, and end the process; if not, jump to step A2;

[0027] Step A2: Initialize the request list batch_data to be empty;

[0028] Step A3: Calculate the number of requests to be taken out using the formula data_num=min(batch_size, request_queue.qsize()), and take the smaller value of batch size batch_size and request queue request_queue length as the number of requests;

[0029] Step A4: Take data_num requests from the request queue request_queue in sequence, and put the request parameters params and object future of each request into the request list batch_data;

[0030] Step A5: Call the model batch inference function to process and obtain the result list results;

[0031] Step A6: traverse the request list batch_data and return the processing result through the future object of each request in the request queue request_queue according to the results in the result list results.

[0032] A further improvement of the present invention is that, in step S201, the sub-dimensions of object understanding include object type, object attributes and object state; the sub-dimensions of knowledge include affordance and world knowledge; the sub-dimensions of spatiotemporal perception include space perception, action perception and time perception; the sub-dimensions of task understanding include task-related objects, task completion conditions and situational reasoning; and the sub-dimensions of planning include closed-loop planning and open-loop planning.

[0033] A further improvement of the present invention is that step S202 includes the following sub-steps:

[0034] Step S2021: Classify the question and answer data in the public dataset according to the capability framework of step S201, and divide the question and answer data into the capability dimensions of the capability framework;

[0035] Step S2022: Using the json module of the Python language, unify the JSON format of all data sets through a Python script;

[0036] Step S2023: forming a standardized JSON data structure, wherein the JSON data structure includes a sample number, a text question, a corresponding image file path, a corresponding video file path, and standard answer information;

[0037] In step S2024, a task threshold is set for each capability dimension, and no more than 500 image question-answer pairs and no more than 200 video question-answer pairs are retained for each capability dimension to balance the image recognition task and the video understanding task.

[0038] A further improvement of the present invention is that, in the evaluation strategy based on the large language model, the formula The answers are scored, where N represents the number of repeated rounds in the evaluation process. Multiple rounds of evaluation are used to reduce the impact of accidental errors on the final results. i represents the round number; M represents the number of sub-dimensions; j represents the sub-dimension number; f LLMi (Gt,A,Rules j ) indicates that in the i-th round, the scoring rules for the j-th sub-dimension are adjusted. j Evaluation function for the standard answer Gt and the candidate answer A.

[0039] A further improvement of the present invention is that, in the evaluation strategy based on the large language model, a multi-round independent scoring mechanism is adopted, each question-answer pair will be evaluated multiple times, and the average value is taken as the final score; and a semantic comparison mechanism is adopted, in which corresponding text scoring rules are given in advance according to the characteristics of each ability dimension to perform semantic comparison and scoring.

[0040] Compared with the prior art, the beneficial effects of the present invention are: it includes a visual embodied question and answer evaluation platform, an embodied question and answer benchmark based on a unified capability framework, and an evaluation strategy based on a large language model; the visual embodied question and answer evaluation platform is a unified platform based on concurrent control and supporting multimodal data, which adopts a combination of concurrent control and distributed architecture to achieve efficient task scheduling and stable performance, and can adapt to the needs of high-load evaluation scenarios; on this basis, the embodied question and answer benchmark based on a unified capability framework realizes a systematic evaluation of the performance of different models through standardized data structure and task design, covering all core capabilities required for embodied question and answer, and comprehensively reflecting the performance of the model in different tasks; further, the evaluation strategy based on a large language model scores the answers based on multiple independent scoring and semantic comparison mechanisms, providing an optimized evaluation strategy for the combination of the visual embodied question and answer evaluation platform and the embodied question and answer benchmark based on a unified capability framework, improving the accuracy and consistency of the scoring, overcoming the limitations of traditional evaluation methods in processing complex answers, and effectively promoting the organic integration and development of multimodal intelligence and embodied question and answer fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a system architecture diagram of a visual embodied question-answering evaluation platform according to an embodiment of the present invention;

[0042] Figure 2 This is a graph showing the results of a performance stress test on a visual embodied question-and-answer evaluation platform according to an embodiment of the present invention;

[0043] Figure 3 A diagram showing the embodied question-answering capability classification of an embodied question-answering benchmark according to an embodiment of the present invention;

[0044] Figure 4 This is a diagram showing an example of a capability framework and sub-dimension question-and-answer (Q&A) of an embodied Q&A benchmark according to an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of an evaluation method for an evaluation strategy based on a large language model according to an embodiment of the present invention;

[0046] Figure 6 This is an example diagram of a front-end direct conversation interface of a visual embodied question-answering evaluation platform according to an embodiment of the present invention;

[0047] Figure 7 This is an example diagram of a front-end evaluation model interface of a visual embodied question-answering evaluation platform according to an embodiment of the present invention;

[0048] Figure 8 This is an example diagram of the front-end large model ranking interface of the visual embodied question-answering evaluation platform according to one embodiment of the present invention;

[0049] Figure 9 This is a radar chart showing the embodied question-answering ability scores of a multimodal large model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In the description of the present invention, if "several" is used, it means more than one; if "plurality" is used, it means more than two; if "greater than," "less than," or "exceeds," it should be understood as excluding the number itself; if "above," "below," or "within" is used, it should be understood as including the number itself. If "first," "second," etc. is used, it should be understood that it is used only to distinguish the names of identical or similar technical features, and should not be understood to imply or indicate the relative importance of the technical features, the number of technical features, or the order of the technical features.

[0051] The preferred embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0052] like Figures 1 to 9 As shown, this embodiment provides a multimodal large-model embodied question-answering evaluation system based on a unified platform and benchmark, including a visual embodied question-answering evaluation platform, an embodied question-answering benchmark based on a unified capability framework, and an evaluation strategy based on a large language model; the visual embodied question-answering evaluation platform is a unified platform based on concurrent control and supports multimodal data; the embodied question-answering benchmark based on a unified capability framework realizes a systematic evaluation of model performance through standardized data structure and task design; the evaluation strategy based on the large language model scores the answers based on multiple independent scoring and semantic comparison mechanisms.

[0053] First, we will introduce the theoretical foundations of the evaluation platform. Distributed systems theory, concurrency control theory, and user-friendly design are key technical foundations of modern evaluation platforms. Distributed systems distribute tasks across multiple nodes and monitor node status using a heartbeat mechanism to ensure stable operation in the event of node failures or high loads. Asynchronous task processing relies on Python's asyncio framework and the data validation tool Pydantic, enabling the system to maintain responsiveness and data accuracy in highly concurrent environments. Uvicorn is an Asynchronous Server Gateway Interface (ASGI) server that provides low latency and high performance for high-concurrency requests. The semaphore mechanism, based on concurrency control theory, limits the number of concurrent tasks, avoiding resource contention and system crashes, ensuring stable operation of the system in complex tasks. Combining semaphores with asynchronous stream processing makes multimodal data processing (such as images, videos, and text) more efficient. User-friendly design, through the Gradio front-end framework, provides multimodal data upload and real-time evaluation feedback, allowing users to intuitively participate in and compare the performance of different models. The collaborative design of front-end and back-end systems improves the system's usability and interactivity, helping users to understand and analyze evaluation results more comprehensively.

[0054] The visual embodied question-answering evaluation platform described in this embodiment is also called the EmbodiedEval platform. It is a visual evaluation platform designed specifically for multimodal large language models, supporting multimodal data input and high-concurrency task processing. The core goal of this visual embodied question-answering evaluation platform is to provide a stable and efficient evaluation environment for the performance of multimodal large models in embodied question-answering tasks, and to ensure that the system maintains high performance even in high-load scenarios. The EmbodiedEval platform supports multiple data modalities, including images, videos, and text. Users can upload these data and interact with the model in real time. The visual embodied question-answering evaluation platform integrates multiple large language models through a unified interface to achieve horizontal comparison and evaluation of different models under the same framework. Users can customize the selection of models and benchmarks during evaluation, and display the results in an intuitive form.

[0055] like Figure 1As shown in the figure, in terms of system architecture, the EmbodiedEval platform preferentially uses Python's FastAPI and uvicorn technology, adopting a distributed architecture of model workers, controllers, and servers. Each model worker (ModelWorker) is responsible for the inference tasks of each model and reports its status in real time through a dynamic heartbeat mechanism; the inference tasks refer to multi-benchmark inference, which is achieved by unifying the data formats of multiple benchmarks and a unified reading interface. The controller (Controller) reasonably allocates tasks based on the load of the workers and promptly transfers load and restores nodes when node failures occur. The server (Server) manages user requests and displays the results on the front-end interface, ensuring that users can intuitively view the evaluation results.

[0056] The visual embodied question-answering evaluation platform described in this embodiment is based on a distributed architecture and supports the simultaneous deployment of multiple different multimodal large models on different nodes. Unlike existing technologies, the visual embodied question-answering evaluation platform described in this embodiment uses a distributed architecture to implement the management and deployment of model working nodes, decoupling model deployment from user requests. Through this design, the system can support the simultaneous deployment of multiple different models on different nodes. Therefore, there is no need to reload the model during each inference process, thereby improving inference efficiency and system flexibility.

[0057] Specifically, the implementation process of the visual embodied question-answering evaluation platform described in this embodiment includes the following steps:

[0058] Step S101: Use the Gradio framework to build the front end of the visual embodied question-answering evaluation platform, including a direct dialogue module, a model evaluation module, and a result analysis module. The direct dialogue module is used to conduct multiple rounds of interaction with the multimodal large model to achieve multimodal data input and real-time feedback. Users can conduct multiple rounds of interaction with the multimodal large model through the direct dialogue module; the model evaluation module selects different multimodal large models and benchmark tasks for combined execution to achieve performance evaluation, and quantitatively analyzes the results using preset evaluation indicators; the result analysis module displays the scores and rankings of each multimodal large model through at least one visual method selected from a table, a bar chart, and a radar chart, helping users to quickly and comprehensively grasp the performance of the model. The framework-based front-end design of this embodiment can improve the interactivity and usability of the platform, allowing users to quickly complete complex evaluation tasks.

[0059] Step S102: Implement the backend of the visual embodied question-answering evaluation platform based on the FastAPI architecture and the uvicorn server. The backend adopts a distributed architecture including a model worker, a controller, and a server. The model worker is responsible for loading and running different multimodal large models and reporting the node status to the controller through a heartbeat mechanism. The controller is used to dynamically allocate inference requests according to the load of the model worker. Specifically, when multiple users initiate requests at the same time, tasks are dynamically allocated according to the shortest queue principle, that is, the next task selects the model node with the least operations to be processed, so as to ensure that the request is responded to in a timely manner, and load transfer and node recovery are performed when a node fails, so as to fully guarantee the stability of the system under high concurrency conditions. The server is responsible for receiving front-end requests and returning inference results, so as to achieve seamless connection between the front-end and the back-end.

[0060] Step S103: Use the LOCUST tool to perform a stress test on the visual embodied question-answering evaluation platform.

[0061] In this embodiment, the controller is used to dynamically allocate inference requests according to the load of the model worker, which can be understood as being used to select a model node (referred to as node) for inference, and the inference process of a single node is implemented by the following Algorithm 1 and Algorithm 2. The node automatic registration and status detection mechanism of this embodiment is derived from the service registration and discovery technology in the distributed architecture, which solves the management complexity brought about by the dynamic increase and decrease of nodes, and enables the system to quickly adapt to changes in the environment. If a service instance is down (Downtime), the controller should perceive it in real time and remove the instance information, which is also called off-hook. The heartbeat mechanism can achieve this function, and the supervision of the node status is mainly completed through two methods: passive detection and active detection.

[0062] The monitoring methods for node status in this embodiment include passive detection and active detection. In the passive detection mode, the working node sends heartbeat messages to the registration center regularly. If no heartbeat message is received within a preset number of consecutive cycles, the registration center will determine that the node has crashed and automatically remove it from the service list. The time interval for regularly sending heartbeat messages to the registration center can be set according to actual conditions and needs, such as once every 5 seconds. The preset number of consecutive cycles can also be set according to actual conditions and needs, such as three consecutive cycles, that is, if no heartbeat message is received within 15 seconds, the registration center will determine that the node has crashed. In the active detection mode, the registration center sends detection messages to all nodes regularly. If no response is received from the node within a preset number of consecutive cycles, the node will be actively removed. Through the design of this mechanism, this embodiment can effectively monitor the status of the node and ensure that when a node fails, the system can perceive it in time and make corresponding adjustments.

[0063] In step S101 of this embodiment, the front end uses an interactive front-end tool to submit various types of input data, view the evaluation feedback in real time, and collaborate with the back end through a unified interface. This interactive design ensures that users can participate in the multimodal evaluation in a natural and convenient way, and collaborates with the back end through a unified interface to ensure the efficiency of the multimodal data evaluation process. In step S102, a unified interface is first defined in a distributed architecture through the FastAPI architecture so that tasks can be distributed and processed between different nodes. Then, through the concurrent connection of the uvicorn server, combined with the asynchronous streaming processing model of the FastAPI architecture, data transmission and processing in high-concurrency scenarios are realized. The asynchronous streaming processing model of the FastAPI architecture is used to split the data processing process into non-blocking tasks to realize the processing of multimodal data, which includes images, text and video.

[0064] During its implementation, the FastAPI architecture, as a modern, fast web framework, plays a central role in asynchronous processing across the entire platform. Developed in Python, FastAPI uses type hints and Starlette in Python 3.7+ as the underlying framework. Combined with Pydantic's data validation capabilities, it makes building high-performance API interfaces both efficient and convenient. Its asynchronous capabilities (based on Python's asyncio) ensure the system's responsiveness when handling highly concurrent requests, which is crucial for platforms that need to handle large numbers of model evaluation tasks in real time. At the same time, the FastAPI architecture's design philosophy supports scalable API development, making it easy to define a unified interface in a distributed architecture so that tasks can be distributed and processed across different nodes.

[0065] The high-performance Asynchronous Server Gateway Interface (ASGI) server is an extension of the traditional Web Server Gateway Interface (WSGI) that supports concurrent processing of asynchronous tasks, which provides good support for modern Web applications. uvicorn is a lightweight ASGI server built on uvloop and httptools, specifically designed to provide high-concurrency and low-latency services. By combining the uvicorn server, FastAPI applications can be deployed efficiently and maintain fast responses in high-concurrency environments. This feature is very suitable for scenarios that need to process multiple evaluation requests simultaneously, allowing each model worker node to maintain a low response time even under heavy task loads.

[0066] In this embodiment, the concurrency control technology is crucial when processing multi-model parallel tasks, and the semaphore control mechanism is a specific application of this theory. The design of semaphores can effectively manage access to shared resources, thereby maintaining stable system performance under high load conditions. Each model working node controls the number of concurrencies through semaphores to ensure that the system will not crash due to resource contention during peak request times. The semaphore is combined with FastAPI's asynchronous streaming processing model to enable the system to cope with data transmission and processing in high-concurrency scenarios. FastAPI's asynchronous streaming processing achieves efficient processing of multimodal data (such as images, text, and videos) by splitting the data processing process into non-blocking tasks, thereby ensuring the system's response speed and stability when faced with large amounts of complex data. Uvicorn provides high-performance concurrent connection support in this process, which forms a good complementary relationship with FastAPI's asynchronous capabilities, helps control resource allocation, and prevents the system from bottlenecking due to high-concurrency requests.

[0067] At the same time, front-end design also plays a crucial role in improving user experience and system interactivity. In multimodal evaluation systems, interactive front-end tools, such as Gradio, allow users to conveniently submit various types of input data and view evaluation feedback in real time. This interactive design ensures that users can participate in multimodal evaluation in a natural and convenient manner. Furthermore, through a unified interface and collaboration with the back-end, the efficiency of the multimodal data evaluation process is ensured. Gradio, as an interactive front-end, integrates with the back-end processing of the FastAPI architecture, allowing users to upload various types of data (such as images, text, and videos) through an intuitive interface and quickly obtain results. This front-end design approach stems from a focus on optimizing user experience, aiming to enhance the user experience and platform usability through the flexibility of multimodal input and the instant visualization of output results. Furthermore, an evaluation result comparison function is integrated into the front-end interaction, allowing users to compare the results of multiple models for the same input, thereby obtaining more objective and accurate evaluation recommendations. This approach, combining data visualization and interactive feedback design concepts, not only improves user understanding and trust in evaluation results, but also enhances system usability.

[0068] During the backend implementation process, a distributed architecture design was adopted to decouple model loading from user request processing, shortening system restart time and improving resource utilization efficiency. The backend architecture of this embodiment supports batch task processing and high-concurrency requests, and implements intelligent scheduling and load balancing in a multi-model environment, ensuring stable system operation and efficient response, and improving the user experience. In addition, the platform adopts a unified interface design to support the integration of more models and the expansion of new benchmarks in the future.

[0069] In general, the visual embodied question-and-answer evaluation platform of this embodiment combines distributed architecture, concurrent control theory, and front-end design, thereby providing a good foundation for the development of the evaluation platform. The use of the FastAPI architecture and the uvicorn server ensures the feasibility of technical implementation. The combination of the two provides a solid foundation for high-concurrency processing, rapid response, and task scheduling, enabling the system to meet the evaluation needs of large multimodal models in an efficient, flexible, and stable manner.

[0070] In step S102 of this embodiment, an asynchronous queue mechanism is used to implement data transmission and processing in high-concurrency scenarios. When multiple users initiate requests simultaneously, tasks are dynamically allocated based on the shortest queue principle; that is, the next task selects the model node with the fewest pending operations to ensure that the request is responded to promptly. Actual testing has shown that the visual embodied question-and-answer evaluation platform described in this embodiment demonstrates high stability and scalability. The asynchronous queue mechanism described in this embodiment is used to implement the asynchronous queue request processing process of the model worker, referred to as Algorithm 1, and includes the following sub-steps:

[0071] Step S1021, global initialization, initialization batch size batch_size = 20, maximum waiting time batch_timeout = 1 second, request queue request_queue is empty;

[0072] Step S1022: Use await to wait for the asynchronous request to complete, obtain the keyword of the asynchronous request, and extract the request parameter params from the asynchronous request;

[0073] Step S1023, creating an object future through the asyncio.Future() function to store the result of the request processing;

[0074] Step S1024, putting the request parameters params and the object future into the request queue request_queue;

[0075] Step S1025, determine whether the length of the request queue request_queue is greater than the set batch size batch_size. If so, create an asynchronous task through the asyncio.create_task() function to process the current batch request (Algorithm 2); if not, jump to step S1026;

[0076] Step S1026, use the await asyncio.wait_for() function to wait for the maximum timeout period batch_timeout. Within the timeout period, determine whether the request queue request_queue is empty. If it is not empty, create an asynchronous task through the asyncio.create_task() function to process the current batch request (Algorithm 2) and jump to step S1027; if it is empty, jump directly to step S1027;

[0077] Step S1027: Use await future to obtain the inference result, and return the inference result encapsulated in JSON format based on the JSONResponse tool.

[0078] This example uses the asyncio.create_task() function to create an asynchronous task to process the current batch of requests. This is referred to as Algorithm 2. The implementation process includes the following steps:

[0079] Step A1: Determine whether the request queue request_queue is empty. If so, directly return an empty result or an exception reminder that the queue is empty, and end the process; if not, jump to step A2;

[0080] Step A2: Initialize the request list batch_data to be empty;

[0081] Step A3: Calculate the number of requests to be taken out using the formula data_num=min(batch_size, request_queue.qsize()), and take the smaller value of batch size batch_size and request queue request_queue length as the number of requests;

[0082] Step A4: Take data_num requests from the request queue request_queue in sequence, and put the request parameters params and object future of each request into the request list batch_data through the put() function;

[0083] Step A5: Call the model batch inference function to process and obtain the result list results;

[0084] Step A6: traverse the request list batch_data and return the processing result through the future object of each request in the request queue request_queue according to the results in the result list results.

[0085] like Figure 2 As shown, this embodiment stress tests the visual embodied question-and-answer evaluation platform. The stress test results show that with 128 concurrent users, the average response time increases from 2.17 seconds to 45.56 seconds, and the system error rate always remains at 0%. Even if the GPU utilization rate is close to 96%, the system can still maintain a high throughput, and the CPU load is kept below 12.4%. Therefore, through the stress test, it can be seen that the response time and throughput test of the visual embodied question-and-answer evaluation platform in this embodiment demonstrates its good scalability to the number of users. No errors occurred during the entire test process. The resource utilization test shows that the GPU undertakes the main computing tasks and becomes the main computing bottleneck of the system when processing a large number of requests. The resource allocation of the visual embodied question-and-answer evaluation platform is reasonable and efficient. Under medium load (16 concurrent users), the number of requests per second of the platform remains at around 3.3, and all requests can be responded to within 5 seconds. Therefore, through Figure 2 The stress test effectively demonstrated the excellent scalability and efficient task scheduling capabilities of this embodiment.

[0086] Step S103 described in this embodiment is used to stress test the performance of the visual embodied question-and-answer evaluation platform. In order to verify the stability and scalability of the system, this embodiment preferably uses the LOCUST tool to conduct rigorous stress testing on the platform, covering different load scenarios from low load to high concurrency environments. The test results show that in a single-user environment, the platform's response time is approximately 2.17 seconds, while in an environment with 16 concurrent users, the response time is still controlled within 8 seconds, and the system error rate always remains at 0%. The platform can still maintain a stable throughput and low error rate under high concurrency, thus verifying its excellent scalability and reliability.

[0087] The embodied question answering benchmark described in this example, based on a unified capability framework, is represented by the UniEQA benchmark, also known as the Unified Embodied Question Answering (UEQA), which is used to systematically evaluate the capabilities of large multimodal language models. The UniEQA benchmark integrates multiple high-quality existing embodied question answering datasets and uses a unified format for capability classification, ensuring data diversity and representativeness.

[0088] like Figure 3 and Figure 4 As shown, this embodiment provides a finely divided unified capability framework for embodied question-answering models, breaking down the core capabilities required for embodied intelligence into five categories: object understanding, knowledge, spatio-temporal perception, task understanding, and planning. These five core capabilities cover all key dimensions of embodied intelligence and provide a systematic basis for the comprehensive evaluation and improvement of embodied models.

[0089] Specifically, the implementation process of the embodied question-answering benchmark based on the unified capability framework described in this embodiment includes the following sub-steps:

[0090] Step S201: Building a unified capability framework based on the five capability dimensions of embodied intelligence: object understanding, knowledge, spatiotemporal perception, task understanding, and planning.

[0091] Step S202 integrates and filters the different datasets, classifies the data in the datasets based on the capability framework of step S201, and unifies the data format to construct an embodied question-answering benchmark. Preferably, redundant data in certain datasets is trimmed and filtered to reduce the data size, significantly improving data processing efficiency and reducing system storage and computing resource consumption.

[0092] like Figure 4As shown, in step S201 of this embodiment, the sub-dimensions of object understanding include object type, object property, and object state. Object type refers to the model's ability to recognize different categories of objects. Object property includes features such as the object's color, shape, and material. Object state refers to the understanding of the object's current state, such as whether the object is stationary, moving, open, or closed. The sub-dimensions of knowledge include affordance and world knowledge. Affordance refers to the purpose and available actions of a specific object, such as a chair that can be sat on or a door that can be opened. World knowledge covers common sense understanding, such as the fact that paper burns and glass is fragile, helping the embodied model make better judgments in complex scenarios. In the spatiotemporal perception dimension, the subdimensions of spatiotemporal perception include spatial perception, action perception, and temporal perception. Spatial perception refers to the model's understanding of the relative positions and distances of objects in the environment. Action perception emphasizes the model's perception and understanding of the currently executed action, such as observing the movement of an object or the manipulation of a character. Temporal perception involves understanding and predicting the sequence of events, ensuring the model understands the temporal logic of the task. Task understanding, on the other hand, includes task-related objects, task completion conditions, and situated reasoning, covering multiple aspects related to the task. Task-related objects refers to the model's ability to identify objects or elements closely related to the current task. Task completion conditions require the model to accurately determine whether the task has been successfully completed. Situated reasoning requires the model to make inferences and judgments based on the current environment and task requirements.The sub-dimensions of planning include closed-loop planning and open-loop planning. Closed-loop planning involves task execution with real-time feedback, and the model generates a plan based on current environmental information. Open-loop planning, on the other hand, generates a complete task plan before task execution and does not rely on environmental feedback.

[0093] like Figure 3 As shown, based on the comprehensive division of the above five core capabilities, this embodiment builds a supporting and comprehensive embodied question-and-answer benchmark on the basis of the visual embodied question-and-answer evaluation platform, and further subdivides it into 13 sub-dimensions to achieve the integration of the embodied question-and-answer benchmark, forming a unified capability division and data format. This benchmark design not only covers the wide range of capabilities required for embodied intelligence, but also ensures the diversity of data and scenarios, laying a solid foundation for the comprehensive evaluation and fine-grained analysis of the model. In this way, this embodiment can establish a standardized evaluation benchmark, so that the performance of different embodied models can be compared horizontally under a unified framework, providing a scientific and unified evaluation standard for the performance of different models in various capability dimensions, and promoting the further development of embodied intelligence technology solutions.

[0094] Preferably, step S202 in this embodiment includes the following sub-steps:

[0095] Step S2021: Classify the question-and-answer data in the public datasets according to the capability framework of step S201, and divide the question-and-answer data into the capability dimensions of the capability framework, so as to map the originally different datasets into the unified embodied question-and-answer capability framework proposed in this embodiment; the public datasets include but are not limited to ALFRED, EgoTaskQA, Ego4D, CharadesEgo, EpicKitchens-100, and Mementos datasets;

[0096] Step S2022: Using the json module of the Python language, a Python script is used to unify the JSON format of all data sets to facilitate subsequent use and understanding;

[0097] Step S2023: forming a standardized JSON data structure, wherein the JSON data structure includes information such as the sample number, the text question, the corresponding image file path, the corresponding video file path, and the standard answer;

[0098] In step S2024, a task threshold is set for each capability dimension, retaining no more than 500 image-question-answer pairs and no more than 200 video-question-answer pairs for each capability dimension to balance the image recognition and video understanding tasks and improve data representativeness. The task threshold refers to the preset number of tasks corresponding to each capability dimension and can be adjusted based on actual conditions and needs.

[0099] The task scenarios in the embodied question-answering benchmark described in this embodiment are highly diverse, covering a wide range of question-answering scenarios, from simple object recognition to complex task planning. For example, in object understanding tasks, the model needs to identify objects of different categories and their attributes; in planning tasks, the model needs to generate a multi-step operation sequence to achieve a specific goal. UniEQA not only ensures the diversity of task scenarios, but also achieves standardization of test data, providing a reliable basis for horizontal comparison of models.

[0100] Based on the realization of a visual embodied question-answering evaluation platform and an embodied question-answering benchmark based on a unified capability framework, this embodiment also proposes an improved evaluation strategy based on a large language model. Since traditional evaluation methods are difficult to effectively handle open-ended answers generated by a multimodal large language model and cannot meet the actual evaluation needs of this embodiment, this embodiment improves the evaluation strategy based on a large language model (LLM), thereby improving the accuracy and adaptability of the evaluation. This evaluation strategy solves the shortcomings of traditional rule evaluation in semantic understanding and is highly consistent with human scoring results. The evaluation function f LLM The process is as follows: first, the information to be evaluated (the standard answer Gt and the candidate answer A to be evaluated) and the scoring rules Rules are input into the large language model in text form. Then the large language model scores the candidate answers in text form based on the received information. Finally, a Python script is used to identify the score given by the large model from the text.

[0101] Different from existing evaluation strategies based on large language models, such as Figure 5As shown, first, the evaluation strategy of this embodiment introduces multiple independent scoring and semantic comparison mechanisms. Each question-answer pair is scored multiple times during evaluation, and the platform takes the average score as the final score to reduce the impact of random errors. Furthermore, unlike existing technologies, this embodiment no longer uses a single prompt. Instead, it tailors specific text prompts for each dimension of the task. Specifically, a corresponding text prompt is pre-set for each dimension, effectively avoiding the evaluation inaccuracies caused by sharing the same prompt across different tasks. This improvement ensures that each task dimension is evaluated according to the most appropriate rules, thereby improving the evaluator's understanding of specific tasks and the accuracy of its evaluation. Second, this embodiment introduces a new answer comparison evaluation mechanism, namely the semantic comparison mechanism. Unlike existing technologies, this embodiment no longer provides the evaluator with a specific question description, but instead directly requires the evaluator to compare and judge between two candidate answers. This method avoids the evaluator's preconceived notions about the question context, thereby reducing the potential risk of misleading bias and subjective judgment. This direct comparison method places greater emphasis on the similarities and differences between answers, allowing the model to make more objective decisions based on semantics and logic. This evaluation method not only improves the objectivity of the evaluation, but also enables the evaluator to more accurately capture the differences and pros and cons of answers when faced with open-ended tasks. In addition, this evaluation strategy also utilizes the contextual understanding capabilities of large language models to judge the rationality of answers in specific situations. In the process of generating complex answers, even if the answer format is different from the standard answer, the semantic evaluation algorithm of this embodiment can accurately judge the correctness of the answer. This flexible evaluation method ensures the objectivity and accuracy of the scoring and provides reliable feedback for model optimization.

[0102] Specifically, in the evaluation strategy based on the large language model in this embodiment, the formula The answers are scored, where N represents the number of repeated rounds in the evaluation process. Multiple rounds of evaluation are used to reduce the impact of accidental errors on the final results. i represents the round number; M represents the number of sub-dimensions, which defaults to 13 in this embodiment; j represents the sub-dimension number; f LLMi (Gt,A,Rules j ) indicates that in the i-th round, the scoring rules for the j-th sub-dimension are adjusted. j The evaluation function for the standard answer Gt and the candidate answer A. The evaluation function will score according to the characteristics of each dimension based on the input of different scoring rules. The answer is scored by this formula to get the score Score LLM , this embodiment can realize multi-round, multi-rule independent evaluation to obtain a more stable and reliable score.

[0103] Scoring rules for each sub-dimension j The rules for setting and adjusting can be adjusted according to actual conditions and needs.

[0104] For example, for the object type sub-dimension, if the answer predicted by the large language model is different from the actual answer, but they are actually different names for the same type of object, such as motherboard and cabinet, a score of 1 is scored. If the answer predicted by the large language model is different from the actual answer but the two are similar, such as armchair and sofa, a score of 0.5 is scored; otherwise (if the answer predicted by the large language model is different from the actual answer and the two are not similar), a score of 0 is scored.

[0105] For example, for the sub-dimension of closed-loop planning, if the plan provided by the large language model matches the answer (the answer predicted by the large language model matches the actual answer), a score of 1 is assigned. If the large language model provides a reasonable plan and contains the content of the answer, a score of 0.5 is assigned. Otherwise, a score of 0 is assigned.

[0106] For example, the spatial perception subdimension focuses on space-related vocabulary. If the answer predicted by the large language model matches the actual answer, a score of 1 is awarded. If the answer predicted by the large language model and the actual answer are similar in meaning, such as "in the upstairs bedroom" and "at the top of the stairs," a score of 0.5 is awarded. Otherwise, a score of 0 is assigned.

[0107] Therefore, in the evaluation strategy based on the large language model of this embodiment, a multi-round independent scoring mechanism is adopted, and each question-answer pair will be evaluated multiple times, and the average value will be taken as the final score; and a semantic comparison mechanism is adopted, and corresponding text scoring rules are given in advance according to the characteristics of each ability dimension to perform semantic comparison and scoring, so that the answers are more reasonable and reliable in specific situations.

[0108] In summary, the visual embodied question-answering evaluation platform described in this embodiment implements a unified platform based on concurrent control and supporting multimodal data. It combines concurrent control with a distributed architecture to achieve efficient task scheduling and stable performance, and can adapt to the needs of high-load evaluation scenarios. Therefore, the visual embodied question-answering evaluation platform (EmbodiedEval platform) can quickly integrate new models and new benchmarks, providing good scalability and technical support for future research and applications. On this basis, the embodied question-answering benchmark based on a unified capability framework realizes a systematic evaluation of the performance of different models through standardized data structures and task designs, covering all the core capabilities required for embodied question-answering, and comprehensively reflecting the performance of the model in different tasks.

[0109] Furthermore, the evaluation strategy based on the large language model described in this embodiment also scores the answers based on multiple independent scoring and semantic comparison mechanisms, providing an optimized evaluation strategy that combines the visual embodied question and answer evaluation platform and the embodied question and answer benchmark based on a unified capability framework, thereby improving the accuracy and consistency of the scoring and overcoming the limitations of traditional evaluation methods in dealing with complex answers.

[0110] Therefore, this embodiment can effectively combine the visual embodied question and answer evaluation platform and the embodied question and answer benchmark based on a unified capability framework, providing a full-process unified and comprehensive multimodal large-model embodied question and answer evaluation system, realizing the embodied question and answer capability evaluation on multiple models, multiple benchmarks and multiple modalities, facilitating researchers to conduct multi-angle model analysis and optimization, and effectively promoting the organic integration and development of multimodal intelligence and embodied question and answer fields.

[0111] In order to illustrate the effects that can be brought about by the above-mentioned embodiment of this embodiment, further explanation will be given below through the multimodal large model embodied question and answer ability test and multimodal large model embodied question and answer ability analysis.

[0112] First, the embodied question-answering ability test of multimodal large models.

[0113] Step B1: The user selects the evaluation settings. Figure 7 When selecting a model and benchmark on the platform's front-end evaluation interface, the user first enters the evaluation page, where all supported multimodal large language models, such as GPT-4o, MiniGPT-4, and LLaVA, are displayed. Users can select the appropriate model based on task requirements and specify the UniEQA benchmark to be tested and its corresponding capability dimension, such as object understanding, spatiotemporal perception, or task planning.

[0114] Step B2: The front-end sends an inference request. After clicking the "Start Inference" button, the front-end uploads the corresponding test data, such as text and images, and confirms the test configuration. The front-end generates the request JSON data for the corresponding model and sends it to the back-end via the HTTP API.

[0115] In step B3, the controller assigns inference requests. Front-end requests are first received by the server, which organizes and packages the requests based on the user's preferences and forwards them to the controller. The controller, as the task scheduling core, manages all incoming requests and, based on the current system load and the status of the model workers, follows the shortest queue principle, assigning tasks to the model worker with the fewest pending requests. Each request is placed in the model worker's asynchronous queue for processing.

[0116] Step B4: The model processes the request. After the Controller dispatches the request to the corresponding Model Worker node, the Model Worker uses the loaded multimodal model to perform inference on the incoming multimodal data. After processing is complete, the model generates predictions and transmits them back to the Controller in JSON format.

[0117] Step B5: Result Return and Storage. After receiving the results from the model worker, the Controller forwards them to the Server. The Server is responsible for storing these results and generating the corresponding result file. The result file contains detailed information about the request, such as the model name, baseline and capability dimensions, and the specific answers generated by the model.

[0118] Step B6, score calculation. After model inference is complete, users can select different evaluation metrics on the interface to evaluate the model's performance. For example, users can choose large language model-based metrics or traditional rule-based metrics. The server compares and analyzes the model's predictions with the standard answers based on the user's selected evaluation method and calculates the final score. The platform uses a multi-round independent scoring mechanism. The system evaluates each question-answer pair multiple times and takes the average of these scores as the final score to reduce errors in individual scoring.

[0119] Throughout the testing process, the system supported concurrent multi-user request processing. The Controller dynamically monitored the ModelWorker load to ensure efficient scheduling and minimize response time. In high-concurrency scenarios, the Controller automatically assigned tasks to idle nodes, ensuring system stability and reliability. The above process fully automated everything from user data upload to model inference and result scoring.

[0120] Second, analysis of the embodied question-answering capabilities of large multimodal models.

[0121] After the scoring calculation is completed, the EmbodiedEval platform displays the evaluation results in a visual way to help users deeply understand the specific performance of each model in different capability dimensions. Figure 7 In the "Evaluation Model" interface shown, the platform presents the model's comprehensive score and performance across various dimensions, including object understanding, knowledge reasoning, spatiotemporal perception, task understanding, and task planning, using dynamically adjusted tables and visual analysis charts (such as bar charts and radar charts). This flexible visualization not only improves data readability but also provides intuitive support for horizontal comparisons between models.

[0122] exist Figure 8 In the "EmbodiedEval Ranking" module shown, users can select specific models, benchmark tasks, and capability items according to task requirements. The table will be adjusted and updated in real time to show the specific scores and rankings of each model under the selected content. Figure 8 and Figure 9 As shown in the results, the GPT-4o model ranked first with an overall score of 49.6, and performed well in knowledge reasoning (69.67), object understanding (43.82), spatiotemporal perception (43.89), and task understanding (44.42), demonstrating its superior ability in embodied question answering tasks. Among other open source models, MiniCPM-V 2.6 and LLaVA-Next-Video ranked second and third with overall scores of 41.51 and 40.97, respectively, demonstrating good embodied reasoning capabilities.

[0123] The table below shows the embodied question-answering scores of various models. GPT-4o performed the best, ranking first in both overall and individual scores. MiniCPM-V (2.6) and LLaVA-NeXT-Video followed closely behind, making them two of the best performing open-source models.

[0124]

[0125] In comparison, while EmbodiedGPT possesses some knowledge reasoning capabilities, with an overall score of 32.56, its performance in other capabilities is relatively weak. LLaVA and InstructBLIP, while having similar overall scores, each have their own strengths and weaknesses. LLaVA excels in knowledge (47.1) and planning (30.2), while InstructBLIP possesses better object understanding (28.63) and spatiotemporal perception (30.51). Furthermore, PLLaVA, VideoLLaVA, and LLaMA-VID-video all performed very similarly, with scores ranging from 27 to 29. These models' task understanding scores (such as LLaMA-VID-video's 19.78) were relatively low, indicating that when faced with complex environments, these models struggle to effectively reason and make decisions based on the corresponding embodied tasks.

[0126] In the "EmbodiedEval Ranking" module, users can select models, capability dimensions, and benchmark tasks, and visually observe the performance differences between models in the form of bar charts or radar charts. For example, in the dimension of object understanding, GPT-4o performs significantly better than other models, while MiniGPT-4 and Llama-vid-image show obvious shortcomings. Therefore, this embodiment comprehensively evaluates multiple large multimodal models (such as GPT-4o, LLaVA, BLIP-2, etc.) based on the proposed embodied question-answering benchmark. The performance of each model has been analyzed and compared in a fine-grained manner, revealing the advantages and disadvantages of each model in different tasks. The combination of comprehensive evaluation, fine-grained analysis, and visual analysis in this embodiment can effectively help researchers discover the pros and cons of models and select appropriate models according to specific needs.

[0127] In addition, users can also Figure 6 The direct dialogue module shown here allows for direct communication with the selected model. This allows for flexible questioning, real-time visualization of the model's output, and qualitative analysis of its characteristics and strengths. These evaluation results allow researchers to optimize the model's weaknesses. For example, for models with insufficient task planning and comprehension capabilities, a strategy combining interactive learning and reinforcement learning can be employed to improve the model's planning capabilities through multiple rounds of task simulation. For models with poor object comprehension capabilities, such as MiniGPT-4 and Llama-vid-image, their object recognition capabilities can be improved by expanding the types of objects and scenarios in the training dataset.

[0128] Therefore, this embodiment provides a multimodal, large-scale embodied question-answering evaluation system based on a unified platform and benchmark. This comprehensive, systematic evaluation and analysis process not only provides strong support for model optimization and fine-tuning, but also offers important guidance for future model development and integration. By combining EmbodiedEval with the UniEQA benchmark, a highly unified and comprehensive embodied question-answering capability evaluation system has been formed. The accompanying evaluation strategies have been optimized, accelerating the effective integration, application, and development of large multimodal models in the embodied domain.

[0129] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark, characterized by: The system includes a visual embodied question-answering evaluation platform, an embodied question-answering benchmark based on a unified capability framework, and an evaluation strategy based on a large language model. The visual embodied question-answering evaluation platform is a unified platform based on concurrent control and supports multimodal data. The embodied question-answering benchmark based on the unified capability framework achieves systematic evaluation of model performance through standardized data structures and task design. The evaluation strategy based on the large language model scores answers based on multiple independent scoring and semantic comparison mechanisms. The implementation process of the visual embodied question-answering evaluation platform includes the following steps: Step S101: Using the Gradio framework, the front end of the visual embodied question-answering evaluation platform is constructed, comprising a direct dialogue module, a model evaluation module, and a result analysis module. The direct dialogue module is used to conduct multiple rounds of interaction with the multimodal large model to implement multimodal data input and real-time feedback. The model evaluation module selects different multimodal large models and benchmark tasks for combined execution to achieve performance evaluation, and quantitatively analyzes the results using preset evaluation indicators. The result analysis module displays the scores and rankings of each multimodal large model through at least one of a table, a bar chart, and a radar chart. Step S102: Implement the backend of the visual embodied question-answering evaluation platform based on the FastAPI architecture and the uvicorn server. The backend adopts a distributed architecture including a model worker, a controller, and a server. The model worker is responsible for loading and running different multimodal large models and reporting node status to the controller through a heartbeat mechanism. The controller is used to dynamically allocate inference requests based on the load of the model worker and perform load transfer and node recovery when a node fails. The server is responsible for receiving front-end requests and returning inference results, thereby achieving docking between the front-end and back-end. Step S103: Use the LOCUST tool to perform a stress test on the visual embodied question-answering evaluation platform; The implementation process of the embodied question-answering benchmark based on the unified capability framework includes the following sub-steps: Step S201: Building a unified capability framework based on the five capability dimensions of embodied intelligence: object understanding, knowledge, spatiotemporal perception, task understanding, and planning. In step S202, different data sets are integrated and screened, the data in the data sets are classified based on the capability framework of step S201, and the data format is unified to construct an embodied question-answering benchmark.

2. The multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark according to claim 1 is characterized in that: The visual embodied question-and-answer evaluation platform is based on a distributed architecture and supports multiple different multimodal large models to be deployed on different nodes at the same time; the node status supervision methods include passive detection and active detection; in the passive detection method, the working node regularly sends heartbeat messages to the registration center. If no heartbeat message is received within a preset number of consecutive cycles, the registration center will determine that the node has crashed and automatically remove it from the service list; in the active detection method, the registration center regularly sends detection messages to all nodes. If no response is received from the node within a preset number of consecutive cycles, the node will be actively removed.

3. The multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark according to claim 1 is characterized in that: In step S101, the front-end uses an interactive front-end tool to submit various types of input data, view evaluation feedback in real time, and collaborate with the back-end through a unified interface.

4. The multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark according to any one of claims 1 to 3, characterized in that: In step S102, a unified interface is first defined in a distributed architecture through the FastAPI architecture so that tasks can be distributed and processed between different nodes; Then, through the concurrent connections of the uvicorn server, combined with the asynchronous streaming processing model of the FastAPI architecture, data transmission and processing in high-concurrency scenarios are achieved; the asynchronous streaming processing model of the FastAPI architecture is used to split the data processing process into non-blocking tasks to realize the processing of multimodal data, which includes images, text and video.

5. The multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark according to claim 4 is characterized in that: In step S102, an asynchronous queue mechanism is used to implement data transmission and processing in high-concurrency scenarios. When multiple users initiate requests simultaneously, tasks are dynamically allocated based on the shortest queue principle. The asynchronous queue mechanism is used to implement the asynchronous queue request processing flow of the model worker, including the following sub-steps: Step S1021, global initialization, initialization batch size batch_size = 20, maximum waiting time batch_timeout = 1 second, request queue request_queue is empty; Step S1022: Use await to wait for the asynchronous request to complete, obtain the keyword of the asynchronous request, and extract the request parameter params from the asynchronous request; Step S1023, creating an object future through the asyncio.Future() function to store the result of the request processing; Step S1024, putting the request parameters params and the object future into the request queue request_queue; Step S1025, determine whether the length of the request queue request_queue is greater than the set batch size batch_size. If so, create an asynchronous task through the asyncio.create_task() function to process the current batch request; if not, jump to step S1026; Step S1026, use the await asyncio.wait_for() function to wait for the maximum timeout period batch_timeout. Within the timeout period, determine whether the request queue request_queue is empty. If it is not empty, create an asynchronous task through the asyncio.create_task() function to process the current batch request, and jump to step S1027; if it is empty, jump directly to step S1027; Step S1027: Use await future to obtain the inference result, and return the inference result encapsulated in JSON format based on the JSONResponse tool.

6. The multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark according to claim 5 is characterized in that: Create an asynchronous task through the asyncio.create_task() function to process the current batch of requests. The implementation process includes the following steps: Step A1: Determine whether the request queue request_queue is empty. If so, directly return an empty result or an exception reminder that the queue is empty, and end the process; if not, jump to step A2; Step A2: Initialize the request list batch_data to be empty; Step A3: Calculate the number of requests to be taken out using the formula data_num=min(batch_size, request_queue.qsize()), and take the smaller value of batch size batch_size and request queue request_queue length as the number of requests; Step A4: Take data_num requests from the request queue request_queue in sequence, and put the request parameters params and object future of each request into the request list batch_data; Step A5: Call the model batch inference function to process and obtain the result list results; Step A6: traverse the request list batch_data and return the processing result through the future object of each request in the request queue request_queue according to the results in the result list results.

7. The multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark according to any one of claims 1 to 3, characterized in that: In step S201, the sub-dimensions of object understanding include object type, object attributes, and object state; the sub-dimensions of knowledge include affordance and world knowledge; the sub-dimensions of spatiotemporal perception include space perception, action perception, and time perception; the sub-dimensions of task understanding include task-related objects, task completion conditions, and situational reasoning; and the sub-dimensions of planning include closed-loop planning and open-loop planning.

8. The multimodal large-model embodied question-answering evaluation system based on a unified platform and benchmark according to any one of claims 1 to 3, characterized in that: The step S202 includes the following sub-steps: Step S2021: Classify the question and answer data in the public dataset according to the capability framework of step S201, and divide the question and answer data into the capability dimensions of the capability framework; Step S2022: Using the json module of the Python language, unify the JSON format of all data sets through a Python script; Step S2023: forming a standardized JSON data structure, wherein the JSON data structure includes a sample number, a text question, a corresponding image file path, a corresponding video file path, and standard answer information; In step S2024, a task threshold is set for each capability dimension, and no more than 500 image question-answer pairs and no more than 200 video question-answer pairs are retained for each capability dimension to balance the image recognition task and the video understanding task.

9. The multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark according to any one of claims 1 to 3, characterized in that: In the evaluation strategy based on the large language model, the formula The answers are scored, where N represents the number of repeated rounds in the evaluation process. Multiple rounds of evaluation are used to reduce the impact of accidental errors on the final results. i represents the round number; M represents the number of sub-dimensions, and j represents the sub-dimension number. Indicates that in the i-th round, the scoring rules for the j-th sub-dimension are adjusted. j Evaluation function for the standard answer Gt and the candidate answer A.

10. The multimodal large-scale embodied question-answering evaluation system based on a unified platform and benchmark according to claim 9 is characterized in that: In the evaluation strategy based on the large language model, a multi-round independent scoring mechanism is adopted. Each question-answer pair will be evaluated multiple times, and the average value will be taken as the final score. A semantic comparison mechanism is also adopted to give corresponding text scoring rules in advance according to the characteristics of each ability dimension for semantic comparison and scoring.

Images