MLLM multi-agent data set construction method, apparatus and device, and program product

Abstract

The invention discloses an MLLM multi-agent data set construction method and device, equipment and a program product, and the method comprises the steps: analyzing a natural language demand text through an MLLM central coordinator, obtaining a structured task list, and carrying out the task distribution; the method comprises the following steps: acquiring an original image through an image processing agent, driving an MLLM central coordinator to return an image attribute description file based on a structured Prompt project, and performing differentiated optimization processing based on quantitative attributes to obtain an optimized image meeting KL divergence constraint and OSR index constraint; annotating the optimized image through a data annotation agent, and performing quality detection on an annotation result through a monitoring agent; and when it is detected that the annotation is abnormal, a structured abnormal report is reported through the monitoring agent, so that the MLLM central coordinator executes a multi-level rescheduling decision and issues a task rescheduling instruction. The method improves the efficiency and quality of data set expansion, and can be applied to the technical field of artificial intelligence.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and program product for constructing an MLLM (Multi-Agent Learning Model) dataset. Background Technology

[0002] In artificial intelligence, particularly computer vision, high-quality labeled datasets are the cornerstone of training and evaluating models. Currently, the construction of real-world image datasets mainly relies on manual collection, annotation, and quality checks—a time-consuming, costly process prone to subjective bias and errors. While some automated or semi-automated data augmentation and synthesis techniques exist, they suffer from significant shortcomings in the following aspects: 1) Rigid Processes vs. Reactive Execution: Existing automation tools generally employ fixed, linear processing pipelines. From image acquisition to preprocessing, feature extraction, and output, the calling relationships between modules are pre-defined. This architecture lacks the ability to dynamically perceive task requirements and data content, cannot adjust the process according to specific circumstances, and has a low level of intelligence.

[0003] 2) Lack of deep semantic understanding of complex image attributes: Traditional methods struggle to effectively model and quantify abstract semantic attributes in real images, such as "slight occlusion," "twilight lighting," and "complex background." Therefore, their so-called "optimization" and "enhancement" are often "one-size-fits-all" operations based on general algorithms, failing to achieve targeted, high-quality processing.

[0004] 3) The process is fragile and prone to "chain breakage": In a fixed pipeline, any semantic "illusion error" at any stage (especially the annotation stage) will cause the error to propagate downstream and contaminate the entire dataset. Furthermore, such systems usually lack the ability to self-diagnose and repair semantic errors. Once a failure occurs, the entire process will be interrupted, heavily relying on manual investigation and intervention, rendering automation ineffective.

[0005] 4) Blindness in data expansion: When expanding the dataset, existing methods are unable to effectively guarantee the consistency between the new data and the original data in the high-dimensional feature space and key attribute distribution, which can easily introduce distribution shift and thus impair the performance of downstream models.

[0006] The above problems urgently need to be addressed. Summary of the Invention

[0007] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0008] Therefore, one objective of this invention is to provide a method for constructing MLLM multi-agent datasets, which improves the efficiency and quality of dataset expansion.

[0009] Another objective of this invention is to provide an apparatus for constructing MLLM multi-agent datasets.

[0010] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a method for constructing an MLLM multi-agent dataset, comprising the following steps: The MLLM central coordinator parses the natural language requirement text input by the user to obtain a structured task list, and then assigns tasks according to the structured task list. The original image is acquired by an image processing agent. The MLLM central coordinator is driven by a structured Prompt project to return the corresponding image attribute description file. The image attribute description file is then subjected to differential optimization processing based on the quantized attributes to obtain an optimized image that satisfies the KL divergence constraint and the OSR index constraint. The optimized image is labeled using a data labeling agent, and the labeling results are quality checked using a monitoring agent. When an anomaly is detected, the monitoring agent reports the corresponding structured anomaly report, which enables the MLLM central coordinator to execute a multi-level rescheduling decision and issue the corresponding task rescheduling instruction. The successfully labeled optimized images are added to the target dataset.

[0011] Furthermore, in one embodiment of the present invention, the step of parsing the natural language request text input by the user to obtain a structured task list, and allocating tasks according to the structured task list, specifically includes: The user's input text representing the natural language requirement is obtained through an interactive interface; The natural language request text is subjected to intent recognition and slot filling to obtain structured intent; The demand analysis agent converts the structured intent into the structured task list. According to the structured task list, the corresponding image processing task is sent to the image processing agent, the data annotation task is sent to the data annotation agent, and the monitoring task is sent to the monitoring agent. The structured task list includes image target type, image scene conditions, dataset OSR index, and output format.

[0012] Furthermore, in one embodiment of the present invention, the step of acquiring the original image and driving the MLLM central coordinator to return the corresponding image attribute description file based on the structured Prompt project specifically includes: The image target type, image scene conditions, and dataset OSR index are determined based on the image processing task issued by the MLLM central coordinator. According to the image target type, obtain the corresponding candidate image from the original image library, and generate the corresponding JSON query template according to the image scene conditions and the dataset OSR index; A structured Prompt instruction is generated based on the candidate image and the JSON query template, and the structured Prompt instruction is uploaded to the MLLM central coordinator. The MLLM central coordinator executes the structured Prompt instruction to obtain the image attribute description file, and then returns the image attribute description file to the image processing agent.

[0013] Furthermore, in one embodiment of the present invention, the differential optimization processing of the image attribute description file based on quantization attributes to obtain an optimized image that satisfies the KL divergence constraint and the OSR index constraint specifically includes: The image attribute description file is parsed to obtain a first image attribute value corresponding to the image scene conditions and a second image attribute value corresponding to the OSR index of the dataset. Based on the first image attribute value, the corresponding image processing tool is called from the preset toolkit to optimize the candidate image and obtain the target image; Calculate whether the KL divergence of the target dataset before and after adding the target image is lower than a preset first threshold; When the KL divergence is lower than the first threshold, calculate the current OSR metric of the target dataset and determine whether the current OSR metric meets the OSR metric of the dataset. When the current OSR metric does not meet the OSR metric of the dataset, the corresponding image processing tool is called from the toolkit according to the second image attribute value to perform targeted enhancement processing on the target image to obtain the optimized image, and the image attribute description file of the optimized image is updated, and then the optimized image is added to the optimized image library; When the current OSR metric meets the OSR metric of the dataset, the target image is added to the optimized image library as the optimized image.

[0014] Furthermore, in one embodiment of the present invention, the step of annotating the optimized image using a data annotation agent specifically includes: In response to the data annotation task issued by the MLLM central coordinator, the data annotation agent obtains the corresponding optimized image and the image attribute description file from the optimized image library; The optimized image is labeled according to the image attribute description file to obtain the labeling results.

[0015] Furthermore, in one embodiment of the present invention, the step of performing quality detection on the annotation results by monitoring the intelligent agent specifically includes: In response to the monitoring task issued by the MLLM central coordinator, the monitoring agent obtains the optimized image and the corresponding annotation result, and determines whether the annotation result matches the optimized image. When the annotation result matches the optimized image, the annotation of the optimized image is determined to be successful; If the annotation result does not match the optimized image, the annotation of the optimized image is determined to be abnormal, and a corresponding structured anomaly report is generated.

[0016] Furthermore, in one embodiment of the present invention, the step of executing multi-level rescheduling decisions and issuing corresponding task rescheduling instructions specifically includes: The error type of the current anomaly is determined based on the structured anomaly report, and it is determined whether the number of anomaly annotations corresponding to the error type within a preset short time period reaches a preset second threshold. When the number of anomaly annotations corresponding to the error type reaches a preset second threshold within a preset short period of time, an upstream parameter adjustment instruction is generated and sent to the image processing agent, so that the image processing agent adjusts the image processing parameters. If the number of anomaly labels corresponding to the error type within a preset short time period does not reach the second threshold, it is determined whether the anomaly labels corresponding to the error type continue to occur within a preset long time period. When the abnormal labels corresponding to the error type continue to occur within a preset long period of time, a task replanning instruction is generated and sent to the data labeling agent, so that the data labeling agent changes the labeling model. When an anomaly label corresponding to the error type occurs by chance within a preset long period of time, a corrective retry instruction is generated and sent to the data labeling agent, so that the data labeling agent re-labels the optimized image according to the corrective retry instruction.

[0017] On the other hand, embodiments of the present invention provide an apparatus for constructing an MLLM multi-agent dataset, comprising: The requirement parsing module is used to parse the natural language requirement text input by the user through the MLLM central coordinator, obtain a structured task list, and allocate tasks according to the structured task list. The optimization processing module is used to acquire the original image through the image processing agent, drive the MLLM central coordinator to return the corresponding image attribute description file based on the structured Prompt project, and perform differential optimization processing on the image attribute description file based on the quantized attributes to obtain an optimized image that satisfies the KL divergence constraint and the OSR index constraint. The annotation module is used to annotate the optimized image through a data annotation agent and to perform quality detection on the annotation results through a monitoring agent. The task rescheduling module is used to report the corresponding structured anomaly report through the monitoring agent when an anomaly is detected, so that the MLLM central coordinator can perform multi-level rescheduling decisions and issue corresponding task rescheduling instructions. The dataset generation module is used to add the successfully labeled optimized images to the target dataset.

[0018] On the other hand, embodiments of the present invention provide an electronic device, the electronic device including a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory, wherein the computer program, when executed by the processor, implements the MLLM multi-agent dataset construction method as described above.

[0019] On the other hand, embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more computer programs, which can be executed by one or more processors to implement the MLLM multi-agent dataset construction method as described above.

[0020] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the MLLM multi-agent dataset construction method as described above.

[0021] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: In this embodiment of the invention, the MLLM central coordinator parses the natural language requirement text input by the user to obtain a structured task list, and allocates tasks according to the structured task list. The image processing agent acquires the original image, and the MLLM central coordinator returns the corresponding image attribute description file based on the structured Prompt project. The image attribute description file is then subjected to differential optimization processing based on quantized attributes to obtain an optimized image that meets the KL divergence constraint and OSR index constraint. The optimized image is labeled by the data labeling agent, and the labeling results are quality checked by the monitoring agent. When an annotation anomaly is detected, the monitoring agent reports the corresponding structured anomaly report, causing the MLLM central coordinator to execute a multi-level rescheduling decision and issue the corresponding task rescheduling instruction to add the successfully labeled optimized image to the target dataset. This invention achieves end-to-end automation from demand input to dataset output through an MLLM central coordinator, an image processing agent, and a data annotation agent, improving the efficiency of dataset expansion. An expansion strategy based on KL divergence constraints and OSR metric constraints ensures statistical consistency of the dataset, improving the quality of dataset expansion. By monitoring the annotation quality and using the MLLM central coordinator to make multi-level rescheduling decisions for abnormal annotations, the risk of single-point failures is reduced, further improving the efficiency and quality of dataset expansion. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating the steps of a method for constructing an MLLM multi-agent dataset according to an embodiment of the present invention; Figure 2 This is a data interaction diagram illustrating the MLLM multi-agent dataset construction method provided in this embodiment of the invention. Figure 3 This is a schematic diagram of the overall process of the MLLM multi-agent dataset construction method provided in the embodiments of the present invention; Figure 4 A flowchart of step A1 provided in an embodiment of the present invention; Figure 5 A flowchart of step A2 provided in an embodiment of the present invention; Figure 6 Another flowchart of step A2 provided in an embodiment of the present invention; Figure 7 A flowchart of step A3 provided in an embodiment of the present invention; Figure 8 Another flowchart of step A3 provided in an embodiment of the present invention; Figure 9 A flowchart of step A4 provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the decision-making process for multi-level rescheduling decisions provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of the MLLM multi-agent dataset construction device provided in an embodiment of the present invention; Figure 12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of the structure of the storage medium provided in an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are only set for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0026] The meanings of some terms used in the description of this invention are as follows: 1. Multi-agent: refers to software entities that operate in parallel and autonomously within a system and possess specific professional functions. The agents in this invention are not simply a collection of functional modules, but rather highly specialized units that, under the unified coordination of MLLM, simulate the collaborative working mode of a human expert team, including "requirements analysis agent, image processing agent, data annotation agent, and monitoring agent".

[0027] 2. MLLM: Large Scale Language Model. In this invention, MLLM plays a core role as a "central coordinator" or "system brain." It not only processes natural language input, but its core responsibility is to act as a structured data processor, capable of parsing structured input instructions (such as a JSON-formatted task list) and generating structured output (such as instructions with correction parameters or decision logs). Its core capabilities are: 1) Deep understanding of natural language requirements; 2) Intelligent decomposition and orchestration of complex tasks into machine-readable task flows containing explicit parameters and execution order; 3) Dynamic task scheduling based on structured agent state reports; 4) Upon receiving structured anomaly feedback, multi-level process replanning based on preset decision logic.

[0028] 3. Cross-modal semantic alignment engine: The core module located within the image processing agent. Its innovation lies in its implementation of a visual attribute quantification method based on structured Prompt engineering. This method does not simply ask the MLLM (Multi-Level Modeling) for questions, but instead executes the following steps: 1) For the image to be analyzed, a JSON template containing predefined key-value pairs is constructed, where the keys represent the visual attributes to be quantified (such as lighting_condition, occlusion_level), and the values ​​are preset types or ranges; 2) The image data and this JSON template are used as input to form a structured Prompt, which is then submitted to the MLLM; 3) The MLLM populates the JSON template based on the image content, thereby outputting machine-readable, structured visual attribute data. This JSON data is the key technical basis for all subsequent intelligent and differentiated processing decisions.

[0029] 4. KL divergence: measures the difference between the feature distribution of newly added samples and the distribution of the original dataset.

[0030] 5. OSR (Occlusion Sample Ratio): This metric is defined as the percentage of samples with occlusion levels greater than or equal to partial occlusion in the total dataset. For example, a system objective might be an OSR > 12.8% to ensure model robustness in high-risk scenarios such as autonomous driving.

[0031] 6. Illusionary errors: These refer to erroneous outputs that do not conform to the semantics of the real scene due to model limitations during automated processing (e.g., mislabeling trees as people). This is a key technical pain point in existing automated processes.

[0032] 7. Self-Healing Mechanism: This refers to a closed-loop intelligent process recovery mechanism driven by MLLM and based on structured anomaly reports, as described in this invention. When the monitoring agent detects an anomaly, it generates and reports a structured report containing fields such as anomaly type, location of occurrence, related data ID, and confidence level. Upon receiving this report, MLLM does not perform a simple "retry" operation, but triggers a multi-level decision logic based on the report content: 1) Diagnosis and Correction Instructions: For low-frequency, occasional errors, retry instructions with specific correction prompts are generated; 2) Upstream Parameter Adjustment: For systemic errors caused by upstream modules (such as image enhancement), instructions are generated to adjust the processing parameters of the upstream modules and backtrack the processing; 3) Process Strategy Reorganization: For persistent errors that are difficult for specific models to handle, the subsequent task allocation strategy is adjusted (such as calling a backup model). This mechanism achieves a leap from "fault capture" to "root cause diagnosis and intelligent repair".

[0033] The MLLM multi-agent dataset construction method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a machine vision-based driver status monitoring method, but is not limited to the above forms.

[0034] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0035] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0036] This invention aims to address the following problems in existing image dataset construction techniques: 1) Rigid automated processes and lack of intelligent scheduling capabilities: Existing automated systems mostly adopt preset linear processing procedures, which cannot understand complex and personalized user needs, nor can they dynamically adjust processing strategies according to real-time changes in data content.

[0037] 2) Superficial image semantic understanding leads to a "one-size-fits-all" approach to optimization: Existing technologies struggle to quantify complex semantic attributes in images, such as occlusion, lighting, and viewpoint, resulting in a lack of specificity and limited effectiveness in subsequent quality optimization and data augmentation operations.

[0038] 3) Poor process robustness and easy to be interrupted due to single point of failure: Especially when dealing with semantic-level "illusion errors", existing automated processes lack effective self-detection and intelligent repair capabilities. The processes are very easy to be interrupted and difficult to self-heal, and rely heavily on manual intervention.

[0039] 4) Data expansion can easily introduce distribution shift: When expanding the dataset, it is difficult to ensure that the newly generated or collected data maintains the same distribution as the original dataset in terms of key attributes and high-dimensional features, which may impair the generalization ability of downstream models.

[0040] like Figure 1 The diagram shown is a flowchart of one step of the MLLM multi-agent dataset construction method provided in an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a method for constructing an MLLM multi-agent dataset, which specifically includes the following steps: A1. The MLLM central coordinator parses the natural language requirement text input by the user to obtain a structured task list, and then assigns tasks according to the structured task list. A2. The original image is obtained through the image processing agent. The MLLM central coordinator driven by the structured Prompt project returns the corresponding image attribute description file. The image attribute description file is then subjected to differential optimization processing based on the quantized attributes to obtain an optimized image that satisfies the KL divergence constraint and the OSR index constraint. A3. Use a data annotation agent to annotate the optimized image, and use a monitoring agent to check the quality of the annotation results; A4. When an anomaly is detected, the monitoring agent reports the corresponding structured anomaly report, which enables the MLLM central coordinator to execute multi-level rescheduling decisions and issue corresponding task rescheduling instructions. A5. Add the successfully labeled optimized images to the target dataset.

[0041] like Figure 2 The diagram shown is a data interaction illustration of the MLLM multi-agent dataset construction method provided in an embodiment of the present invention, as detailed below: 1. User input in natural language; 2. After the MLLM central coordinator deeply analyzes the natural language requirements, the requirement analysis agent generates a structured task list containing quantifiable objectives (such as OSR metrics) and issues the architecture task instructions to the image processing agent. 3. After the image processing agent obtains the image from the original image library, it uses its internal cross-modal semantic alignment engine to generate a JOSN query template using structured Prompt engineering, and sends the image and the JOSN query template together to the MLLM central coordinator. 4. The MLLM central coordinator generates a JSON file describing the image attributes and distributes it to the image processing agent; 5. The image processing agent dynamically calls the most suitable tools from the toolkit based on the issued image attribute description file to perform differentiated optimization (such as rain simulation and low light enhancement), and performs targeted enhancement through dual constraints of KL divergence and OSR to ensure data quality and distribution consistency. 6. Labeling and Monitoring: The data labeling agent labels the optimized images, while the monitoring agent performs parallel detection processes, especially for semantic-level "illusion errors". Once an anomaly is detected, the monitoring agent reports a structured anomaly report. 7. Intelligent self-healing and replanning: After receiving a structured anomaly report, the MLLM central coordinator starts a multi-level decision engine. Based on the severity and pattern of the error, it issues instructions such as corrective retry, upstream parameter adjustment, or task replanning. The process returns to the scheduling stage, realizing intelligent self-healing.

[0042] After all tasks (including replanned tasks) are completed and pass quality checks, the system outputs the final high-quality dataset.

[0043] This invention achieves end-to-end automation from demand input to dataset output through an MLLM central coordinator, an image processing agent, and a data annotation agent, improving the efficiency of dataset expansion. An expansion strategy based on KL divergence constraints and OSR metric constraints ensures statistical consistency of the dataset, improving the quality of dataset expansion. By monitoring the annotation quality and using the MLLM central coordinator to make multi-level rescheduling decisions for abnormal annotations, the risk of single-point failures is reduced, further improving the efficiency and quality of dataset expansion.

[0044] like Figure 4 The diagram shown is a flowchart of step A1 provided in an embodiment of the present invention. (Refer to...) Figure 4 As an optional implementation, the natural language requirement text input by the user is parsed to obtain a structured task list, and tasks are assigned according to the structured task list, specifically including: A11. Obtain the user's input natural language request text through the interactive interface; A12. Perform intent recognition and slot filling on the natural language demand text to obtain structured intent; A13. The demand analysis agent converts structured intents into a structured task list. A14. Based on the structured task list, issue the corresponding image processing tasks to the image processing agent, and issue data annotation tasks to the data annotation agent and monitoring tasks to the monitoring agent. The structured task list includes image target type, image scene conditions, dataset OSR metrics, and output format.

[0045] Specifically, users input their natural language text requirements through the system's interactive interface. The MLLM central coordinator receives the text and, through its built-in Natural Language Understanding (NLU) module, performs intent recognition and slot filling to structure the ambiguous requirements, resulting in a structured task list. Then, it sends the corresponding image processing tasks to the image processing agent, data annotation tasks to the data annotation agent, and monitoring tasks to the monitoring agent.

[0046] like Figure 5 The diagram shown is a flowchart of step A2 provided in an embodiment of the present invention. (Refer to...) Figure 5 As an optional implementation, the original image is acquired, and the MLLM central coordinator, driven by the structured Prompt project, returns the corresponding image attribute description file, which specifically includes: A20. Determine the image target type, image scene conditions, and dataset OSR index based on the image processing task issued by the MLLM central coordinator. A21. Obtain the corresponding candidate images from the original image library according to the image target type, and generate the corresponding JSON query template according to the image scene conditions and the dataset OSR index; A22. Generate structured Prompt instructions based on candidate images and JSON query templates, and upload the structured Prompt instructions to the MLLM central coordinator; A23. The MLLM central coordinator executes the structured Prompt instruction to obtain the image attribute description file, and then returns the image attribute description file to the image processing agent.

[0047] Specifically, this step is a cross-modal visual attribute quantization step based on structured Prompt engineering, executed by an image processing agent, and includes: 1) Receive the image to be processed from the central coordinator; 2) Generate a JSON template containing corresponding keys and null values ​​based on a preset set of attributes (such as occlusion, lighting, complexity, etc.); 3) Combine the image data with the JSON template to form a structured Prompt and send it to the MLLM central coordinator, which is at the core. 4) Receive JSON data returned by the MLLM central coordinator, which has been filled in according to the image content, thereby achieving accurate and structured quantification of complex visual attributes of the image.

[0048] like Figure 6 The diagram shown is another flowchart of step A2 provided in an embodiment of the present invention. (Refer to...) Figure 6 As an optional implementation, the image attribute description file is further subjected to differential optimization processing based on quantization attributes to obtain an optimized image that satisfies the KL divergence constraint and the OSR index constraint, specifically including: A24. Parse the image attribute description file to obtain the first image attribute value corresponding to the image scene conditions and the second image attribute value corresponding to the OSR index of the dataset. A25. Based on the first image attribute value, call the corresponding image processing tool from the preset toolkit to optimize the candidate image and obtain the target image; A26. Calculate whether the KL divergence of the target dataset before and after adding the target image is lower than the preset first threshold; A27. When the KL divergence is lower than the first threshold, calculate the current OSR index of the target dataset and determine whether the current OSR index meets the dataset OSR index. A28. When the current OSR metric does not meet the OSR metric of the dataset, the corresponding image processing tool is called from the toolkit according to the second image attribute value to perform targeted enhancement processing on the target image to obtain an optimized image, and the image attribute description file of the optimized image is updated, and then the optimized image is added to the optimized image library. A29. When the current OSR metric meets the dataset OSR metric, add the target image as an optimized image to the optimized image library.

[0049] Specifically, this step involves differential processing based on quantized attributes, dynamic tool invocation, and incremental data expansion and quality control based on dual constraints, executed by the image processing agent, and includes: 1) Parse the JSON data generated in the previous step; 2) Based on the value of a specific attribute in the JSON (e.g., if lighting_condition is low_light), dynamically select and call the most suitable processing tool from the toolkit (such as the "low light enhancement" algorithm instead of the general sharpening algorithm), and set the corresponding parameters; 3) Global Distribution Consistency Constraint: Before adding a newly processed image sample to the dataset, calculate the KL divergence between the feature distribution of the entire dataset after the addition and the distribution of the original baseline dataset. Only when the KL divergence is lower than a preset threshold is the sample initially accepted to avoid distribution shift.

[0050] 4) Key Scene Coverage Constraints: Based on the generated JSON data, the proportion of samples in the dataset that meet specific challenge attributes (such as occlusion_level>0.5) (i.e., OSR metric) is calculated in real time. If the OSR is lower than the target value, a targeted enhancement mechanism is triggered: the "occlusion simulation" tool is actively called to process images with low occlusion_level in order to increase the number of key samples in a targeted manner until the OSR reaches the target.

[0051] like Figure 7 The diagram shown is a flowchart of step A3 provided in an embodiment of the present invention. (Refer to...) Figure 7 As an optional implementation, the optimized image is further annotated by a data annotation agent, specifically including: A31. In response to the data annotation task issued by the MLLM central coordinator, the data annotation agent obtains the corresponding optimized image and image attribute description file from the optimized image library. A32. Annotate the optimized image according to the image attribute description file to obtain the annotation results.

[0052] like Figure 8 The diagram shown is another flowchart of step A3 provided in an embodiment of the present invention. (Refer to...) Figure 8As an optional implementation, the annotation results are further subjected to quality checks by a monitoring agent, which specifically includes: A33. In response to the monitoring task issued by the MLLM central coordinator, the optimized image and the corresponding annotation results are obtained through the monitoring agent, and it is determined whether the annotation results match the optimized image. A34. When the annotation results match the optimized image, the optimized image annotation is confirmed to be successful; A35. When the annotation results do not match the optimized image, determine that the annotation of the optimized image is abnormal and generate the corresponding structured anomaly report.

[0053] like Figure 9 The diagram shown is a flowchart of step A4 provided in an embodiment of the present invention. (Refer to...) Figure 9 As an optional implementation, a multi-level rescheduling decision is performed, and corresponding task rescheduling instructions are issued, specifically including: A41. Determine the error type of the current anomaly label based on the structured anomaly report, and determine whether the number of anomaly labels corresponding to the error type within a preset short time period has reached the preset second threshold. A42. When the number of anomaly annotations corresponding to the error type reaches the preset second threshold within a preset short time period, an upstream parameter adjustment instruction is generated and sent to the image processing agent, so that the image processing agent adjusts the image processing parameters. A43. If the number of anomaly annotations corresponding to the error type within a preset short time period does not reach the second threshold, determine whether the anomaly annotations corresponding to the error type within a preset long time period continue to occur. A44. When the abnormal annotations corresponding to the error type continue to occur within a preset long time period, a task replanning instruction is generated and sent to the data annotation agent, so that the data annotation agent can change the annotation model. A45. When an anomaly label corresponding to an error type occurs by chance within a preset long time period, a corrective retry instruction is generated and sent to the data labeling agent, so that the data labeling agent can re-label the optimized image according to the corrective retry instruction.

[0054] Specifically, this step is a closed-loop intelligent self-healing step based on structured anomaly reporting, which is executed collaboratively by the monitoring agent and the MLLM central coordinator, and includes: 1) Monitor the intelligent agent to detect errors in the annotation results in real time, especially semantic illusion errors, and generate a JSON-formatted anomaly report containing fields such as error type, image ID, and error content; 2) After receiving the report, the MLLM central coordinator makes a judgment based on its internal decision rule base (see example...). Figure 10 (The decision-making process shown) Rule 1 (Random Errors): If it is a single, low-confidence error, a corrective retry instruction with specific prompts such as "Please note the difference between 'pedestrians' and 'pillar objects'" is issued to the data labeling agent.

[0055] Rule 2 (Systematic Error): If multiple similar errors are received within a short period of time, it is determined that the upstream image enhancement is excessive. An upstream parameter adjustment instruction is issued to the image processing agent to "reduce the intensity of the 'low light enhancement' algorithm by 20% and backtrack the processing of the most recent 50 images".

[0056] Rule 3 (Capability Boundary Error): If a certain type of error (such as identifying fire hydrants) continues to occur, a task replanning instruction will be issued to transfer all images that will be identified as containing "fire hydrant" in the future to a backup annotation model that is slower but more accurate.

[0057] like Figure 3 The diagram shown is a schematic representation of the overall process of the MLLM multi-agent dataset construction method provided in this embodiment of the invention. The following is a detailed explanation in conjunction with... Figure 3 A specific embodiment of the present invention will be described in detail below.

[0058] An autonomous driving algorithm engineer proposed a dataset construction requirement through the system's natural language interface: "Build a dataset for my object detection model, focusing on enhancing samples where 'vehicles' and 'pedestrians' are partially occluded under 'nighttime rainy' conditions. The annotation format should be YOLOv8." Phase 1: Task Definition (S101-S103) S101: User Request Input Users input their natural language text requirements through the system's interactive interface.

[0059] S102: MLLM Central Coordinator Resolution Requirements The system's core MLLM central coordinator receives the text. It then uses its built-in Natural Language Understanding (NLU) module to perform intent recognition and slot filling, structuring the ambiguous requirements.

[0060] Intent recognition: object_detection_dataset_creation (object detection dataset creation) Slot filling: target_classes (image target type): [vehicles, pedestrians]; scenario_conditions (image scene conditions): [Night, Rainy Day]; challenge_focus (OSR metric for dataset): partial occlusion; output_format: YOLOv8.

[0061] S103: Requirements Analysis Agent Generates Structured Task List The MLLM central coordinator passes the parsed structured intent to the requirements analysis agent. This agent transforms it into a list of machine-executable tasks (JSON format) containing clear, quantifiable objectives. Crucially, it transforms the vague requirement of "partial occlusion" into a specific system performance metric, min_osr_target (minimum occluded sample ratio target), as follows: { "task_id": "AD_Task_001", "task_type":"object_detection_dataset", "target_classes":["car", "pedestrian"], "enhancement_focus": { "scenario":["night", "rainy"], "challenge": "occlusion" "min_osr_target": 0.20 / / Set the target: at least 20% of the samples in the dataset need to be occluded samples. }, "output_format": "yolov8" Phase Two: Image Processing and Enhancement (S104-S111) S104: MLLM Dynamic Task Scheduling The MLLM central coordinator begins dynamic scheduling based on the generated task list. It first issues instructions to the image processing agent: to begin retrieving images from the raw image database that contain vehicles or pedestrians and whose metadata includes nighttime and rainy weather labels.

[0062] S105: Obtain the original image The image processing agent executes instructions to obtain a batch of candidate images from the original image library.

[0063] S106: [Core] Cross-modal semantic alignment and attribute quantization For each acquired image (e.g., img_0123.jpg, a clear nighttime street view), the cross-modal semantic alignment engine within the image processing intelligence is activated, and the following quantization method based on structured Prompt engineering is executed: 1) Generate JSON query template: The engine generates a JSON template containing predefined keys but with null values; 2) Constructing a structured Prompt: The engine combines the image data with this template to form a structured, imperative Prompt for MLLM, the content of which is as follows: "Instructions: Analyze the input image and, based on the visual content, accurately fill in all values ​​in the following JSON template. For 'lighting_condition', choose from ['bright', 'dim', 'low_light']; for 'weather', choose from ['clear', 'rainy', 'foggy']; for 'occlusion_level', estimate the average occlusion percentage of the target object in the image and output a floating-point number between 0.0 and 1.0." Template to be filled: {"attributes": {"lighting_condition": null, "weather": null,"occlusion_level": null}}; 3) Obtaining quantification results: The MLLM central coordinator executes this instruction and returns a completed JSON description, transforming complex visual attributes into precise, machine-readable data. { "image_id": "img_0123.jpg", "attributes": { "lighting_condition":"low_light", "weather": "clear", / / MLLM determines that there is no rain in this image. "occlusion_level": 0.1 / / MLLM determines that the occlusion level of this image is low. }} S107: Attribute-based Differentiated Quality Optimization and Tool Invocation The image processing agent performs targeted processing based on the JSON description generated by S106: Scenario 1 (Lighting Processing): Due to "lighting_condition": "low_light", the agent dynamically calls the "low light enhancement" algorithm in the image enhancement toolkit instead of the general sharpening algorithm; Scenario 2 (Weather Simulation): Since the task requirement is rainy, but the image's "weather" property is clear, the agent calls the "Rainy Day Effect Simulation" tool in the toolkit to add realistic rain streaks and reflection effects to the image.

[0064] S109 & S110: Incremental Expansion and Targeted Enhancement Based on Dual Constraints When a new image has been processed and is ready to be added to the enhanced image library, the image processing agent executes a dual-constraint decision algorithm: 1) Global distribution constraint check: Calculate the change in KL divergence of the entire dataset after adding the image. Determine if the KL divergence is within a preset threshold, satisfying the distribution consistency requirement; 2) Key Scenario Coverage Check: Check the OSR metric of the current dataset. Assume the current OSR is 18.5%, which is lower than the 20% target set in S103; 3) Targeted Enhancement Decision: If the OSR is not up to standard, the system triggers a targeted enhancement mechanism. Based on the "occlusion_level": 0.1 information in the JSON description of img_0123.jpg, it determines that the image is a suitable target for OSR improvement. Therefore, it actively calls the "occlusion simulation" tool in the toolkit to randomly add foreground occluders to the image. After processing, it calls S106 again to update the JSON description (at this point, occlusion_level may become 0.6), and then adds it back to the image library.

[0065] S111: Generate the final JSON description and output the optimized image. After completing all processing, the image processing agent outputs the optimized image to the enhanced image library, along with a final updated structured JSON description.

[0066] Phase 3: Data Labeling (S112) S112: Automated annotation The MLLM central coordinator issues instructions to the data annotation agent, which retrieves optimized images and their JSON descriptions from the image library, performs automatic annotation, and generates annotation results.

[0067] Phase Four: Real-time Monitoring and Self-Healing (S112-S116) S113: [Core] Monitor the intelligent agent to detect anomalies and generate structured reports. The monitoring agent monitors the process in parallel and in real time. It randomly selects the annotation results for img_0123.jpg and finds an anomaly in an object labeled "pedestrian," so it initiates a verification with the MLLM central coordinator. The MLLM central coordinator returns "No, the object is more like a fire hydrant," determining that an "illusion error" has occurred. The monitoring agent immediately generates and reports a structured JSON anomaly report: { "report_id": "ERR_1024", "error_type":"hallucination", "agent_id":"annotation_agent_F", "details": { "image_id": "img_0123.jpg", "mis-labeled_object_bbox":[450, 600, 480, 680], "incorrect_label":"pedestrian", "suspected_correct_label":"fire_hydrant" }} At this point, the process enters the "monitoring and self-healing" decision branch.

[0068] S115: [Core] MLLM central coordinator triggers multi-level intelligent rescheduling (self-healing) Upon receiving the aforementioned JSON error report, the MLLM central coordinator activates its internal multi-level decision engine to make intelligent decisions based on the context of the error report: Decision 1 (Minor Error - Corrective Retry): If this is the only such error in the recent past, MLLM judges it as an isolated incident. It issues a retry instruction with precise correction information to the data labeling agent: {"action": "retry_annotation", "image_id": "img_0123.jpg", "guidance": "Please pay special attention to distinguish between 'pedestrians' and 'pillar-shaped objects'."} Decision 2 (Systemic Problem - Upstream Parameter Adjustment): If multiple similar illusionary errors caused by images processed with "low-light enhancement" are received within a short period, MLLM determines that the upstream image enhancement is excessive. It will issue an upstream parameter adjustment and backtracking instruction to the image processing agent: {"action":"adjust_and_reprocess", "algorithm":"low_light_enhancement", "parameter_adjustment":{"strength":"-20%"}, "reprocess_image_ids":[...]} After replanning, the process returns to S104, and the MLLM central coordinator continues to execute task scheduling based on the new system state and strategy, thereby achieving closed-loop intelligent self-healing.

[0069] S114: Quality inspection passed If no error is detected in S113, or all replanned tasks are executed successfully, the process proceeds to S114.

[0070] S116: Output the final dataset Once all tasks (including replanned tasks) have been successfully executed and passed the S114 quality check, the system will package and output a high-quality, robust final dataset that meets all user requirements.

[0071] The method flow of the embodiments of the present invention has been described above. It is understood that the embodiments of the present invention achieve end-to-end automation from demand input to dataset output through an MLLM central coordinator, an image processing agent, and a data annotation agent, thereby improving the efficiency of dataset expansion; through an expansion strategy based on KL divergence constraints and OSR index constraints, the statistical consistency of the dataset is ensured, improving the quality of dataset expansion; by monitoring the annotation quality through an agent, and by using the MLLM central coordinator to make multi-level rescheduling decisions for abnormal annotations, the risk of single-point failure is reduced, further improving the efficiency and quality of dataset expansion.

[0072] Compared with the prior art, the embodiments of the present invention have the following advantages: 1) Achieves extremely high levels of automation and process flexibility: By introducing MLLM as a central coordinator, the system can understand and execute complex natural language instructions, realizing end-to-end automation from demand input to dataset output. The dynamic scheduling mechanism breaks the constraints of traditional linear processes, enabling the system to intelligently adapt to different task requirements.

[0073] 2) Significantly improved dataset quality and real-world simulation capabilities: The innovative cross-modal semantic alignment engine enables deep quantification of complex image attributes, making subsequent optimization operations highly targeted. The constructed dataset can better simulate the complexity of real-world scenes and significantly improve the generalization ability of downstream models.

[0074] 3) It endows the system with unprecedented robustness and self-healing capabilities: The intelligent self-healing closed loop of "monitoring-feedback-replanning" can proactively detect and repair process failures at the semantic level, significantly reducing the risk of single point of failure, solving the pain point of traditional automation tools relying on manual intervention due to failure interruption, and ensuring the continuity and stability of the construction process.

[0075] 4) Ensures the effectiveness and distribution consistency of data expansion: Through the incremental expansion strategy under the dual constraints of quality awareness and KL divergence, this invention effectively avoids distribution shift while efficiently expanding the dataset, ensuring the statistical consistency of the dataset, thereby improving the training accuracy of downstream models and outperforming traditional expansion methods.

[0076] like Figure 11 The diagram shown is a structural schematic of the MLLM multi-agent dataset construction device provided in an embodiment of the present invention. (Refer to...) Figure 11 This invention provides an apparatus for constructing an MLLM multi-agent dataset, comprising: The requirement parsing module is used to parse the natural language requirement text input by the user through the MLLM central coordinator, obtain a structured task list, and allocate tasks according to the structured task list. The optimization processing module is used to acquire the original image through the image processing agent, drive the MLLM central coordinator based on the structured Prompt project to return the corresponding image attribute description file, and perform differential optimization processing on the image attribute description file based on the quantized attributes to obtain an optimized image that meets the KL divergence constraint and OSR index constraint. The annotation module is used to annotate optimized images through a data annotation agent and to perform quality checks on the annotation results through a monitoring agent. The task rescheduling module is used to detect labeled anomalies, report the corresponding structured anomaly report through the monitoring agent, and enable the MLLM central coordinator to perform multi-level rescheduling decisions and issue corresponding task rescheduling instructions. The dataset generation module is used to add successfully labeled optimized images to the target dataset.

[0077] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned MLLM multi-agent dataset construction method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0079] like Figure 12 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of the present invention. (Refer to...) Figure 12 This invention provides an electronic device, comprising: The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1202 can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the MLLM multi-agent dataset construction method of the embodiments of this invention. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0080] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0081] like Figure 13 The diagram shown is a structural schematic of the storage medium provided in an embodiment of the present invention. (Refer to...) Figure 13 The present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 1301, which can be executed by one or more processors to implement the above-described MLLM multi-agent dataset construction method.

[0082] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0083] This invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the above-described MLLM multi-agent dataset construction method.

[0084] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0085] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0087] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0088] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0089] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0090] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0092] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0093] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0094] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0095] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0096] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for constructing an MLLM multi-agent dataset, characterized in that, Includes the following steps: The MLLM central coordinator parses the natural language requirement text input by the user to obtain a structured task list, and then assigns tasks according to the structured task list. The original image is acquired by an image processing agent. The MLLM central coordinator is driven by a structured Prompt project to return the corresponding image attribute description file. The image attribute description file is then subjected to differential optimization processing based on the quantized attributes to obtain an optimized image that satisfies the KL divergence constraint and the OSR index constraint. The optimized image is labeled by a data labeling agent, and the labeling results are quality checked by a monitoring agent. When an anomaly is detected, the monitoring agent reports the corresponding structured anomaly report, which enables the MLLM central coordinator to execute a multi-level rescheduling decision and issue the corresponding task rescheduling instruction. The successfully labeled optimized images are added to the target dataset.

2. The method for constructing an MLLM multi-agent dataset according to claim 1, characterized in that, The process of parsing the natural language request text input by the user to obtain a structured task list, and then allocating tasks according to the structured task list, specifically includes: The user's input text representing the natural language requirement is obtained through an interactive interface; The natural language request text is subjected to intent recognition and slot filling to obtain structured intent; The demand analysis agent converts the structured intent into the structured task list. According to the structured task list, the corresponding image processing task is sent to the image processing agent, the data annotation task is sent to the data annotation agent, and the monitoring task is sent to the monitoring agent. The structured task list includes image target type, image scene conditions, dataset OSR index, and output format.

3. The method for constructing an MLLM multi-agent dataset according to claim 1, characterized in that, The process of acquiring the original image, and driving the MLLM central coordinator based on the structured Prompt project to return the corresponding image attribute description file, specifically includes: The image target type, image scene conditions, and dataset OSR index are determined based on the image processing task issued by the MLLM central coordinator. According to the image target type, obtain the corresponding candidate image from the original image library, and generate the corresponding JSON query template according to the image scene conditions and the dataset OSR index; A structured Prompt instruction is generated based on the candidate image and the JSON query template, and the structured Prompt instruction is uploaded to the MLLM central coordinator. The MLLM central coordinator executes the structured Prompt instruction to obtain the image attribute description file, and then returns the image attribute description file to the image processing agent.

4. The method for constructing an MLLM multi-agent dataset according to claim 3, characterized in that, The differential optimization processing of the image attribute description file based on quantization attributes to obtain an optimized image that satisfies the KL divergence constraint and the OSR index constraint specifically includes: The image attribute description file is parsed to obtain a first image attribute value corresponding to the image scene conditions and a second image attribute value corresponding to the OSR index of the dataset. Based on the first image attribute value, the corresponding image processing tool is called from the preset toolkit to optimize the candidate image and obtain the target image; Calculate whether the KL divergence of the target dataset before and after adding the target image is lower than a preset first threshold; When the KL divergence is lower than the first threshold, calculate the current OSR metric of the target dataset and determine whether the current OSR metric meets the OSR metric of the dataset. When the current OSR metric does not meet the OSR metric of the dataset, the corresponding image processing tool is called from the toolkit according to the second image attribute value to perform targeted enhancement processing on the target image to obtain the optimized image, and the image attribute description file of the optimized image is updated, and then the optimized image is added to the optimized image library; When the current OSR metric meets the OSR metric of the dataset, the target image is added to the optimized image library as the optimized image.

5. The method for constructing an MLLM multi-agent dataset according to claim 1, characterized in that, The step of labeling the optimized image using a data labeling agent specifically includes: In response to the data annotation task issued by the MLLM central coordinator, the data annotation agent obtains the corresponding optimized image and the image attribute description file from the optimized image library; The optimized image is labeled according to the image attribute description file to obtain the labeling results.

6. The method for constructing an MLLM multi-agent dataset according to claim 1, characterized in that, The quality inspection of the annotation results by the monitoring agent specifically includes: In response to the monitoring task issued by the MLLM central coordinator, the monitoring agent obtains the optimized image and the corresponding annotation result, and determines whether the annotation result matches the optimized image. When the annotation result matches the optimized image, the annotation of the optimized image is determined to be successful; If the annotation result does not match the optimized image, the annotation of the optimized image is determined to be abnormal, and a corresponding structured anomaly report is generated.

7. A method for constructing an MLLM multi-agent dataset according to any one of claims 1 to 6, characterized in that, The process of executing multi-level rescheduling decisions and issuing corresponding task rescheduling instructions specifically includes: The error type of the current anomaly is determined based on the structured anomaly report, and it is determined whether the number of anomaly annotations corresponding to the error type within a preset short time period reaches a preset second threshold. When the number of anomaly annotations corresponding to the error type reaches a preset second threshold within a preset short period of time, an upstream parameter adjustment instruction is generated and sent to the image processing agent, so that the image processing agent adjusts the image processing parameters. If the number of anomaly labels corresponding to the error type does not reach the second threshold within a preset short time period, it is determined whether the anomaly labels corresponding to the error type continue to occur within a preset long time period. When the abnormal labels corresponding to the error type continue to occur within a preset long period of time, a task replanning instruction is generated and sent to the data labeling agent, so that the data labeling agent changes the labeling model. When an anomaly label corresponding to the error type occurs by chance within a preset long period of time, a corrective retry instruction is generated and sent to the data labeling agent, so that the data labeling agent re-labels the optimized image according to the corrective retry instruction.

8. A device for constructing an MLLM multi-agent dataset, characterized in that, include: The requirement parsing module is used to parse the natural language requirement text input by the user through the MLLM central coordinator, obtain a structured task list, and allocate tasks according to the structured task list. The optimization processing module is used to acquire the original image through the image processing agent, drive the MLLM central coordinator to return the corresponding image attribute description file based on the structured Prompt project, and perform differential optimization processing on the image attribute description file based on the quantized attributes to obtain an optimized image that satisfies the KL divergence constraint and the OSR index constraint. The annotation module is used to annotate the optimized image through a data annotation agent and to perform quality detection on the annotation results through a monitoring agent. The task rescheduling module is used to report the corresponding structured anomaly report through the monitoring agent when an anomaly is detected, so that the MLLM central coordinator can perform multi-level rescheduling decisions and issue corresponding task rescheduling instructions. The dataset generation module is used to add the successfully labeled optimized images to the target dataset.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When the computer program is executed by the processor, it implements the MLLM multi-agent dataset construction method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the MLLM multi-agent dataset construction method as described in any one of claims 1 to 7.

Images