Model optimization method and device based on visual semantic data enhancement and storage medium

Abstract

The invention provides a model optimization method based on visual semantic data enhancement, which is used for improving model performance under a visual semantic task. The method focuses on the influence of object semantic information in a scene on a model, firstly, the model performance is evaluated by utilizing a preset data set, key semantic information is summarized by utilizing a large language model according to the model performance and scene data, then the semantic information is injected into newly generated data, and then new model training data is generated. And a continuously improved cycle is formed. Compared with a traditional data enhancement method, the method has the advantages that the data enhancement process is matched with the model defects, a training scene more suitable for improving the model performance is provided in an automatic form, and the method can be used for a plurality of personal tasks such as visual semantic navigation and visual language navigation.

Description

Technical Field

[0001] The present invention belongs to the field of embodied intelligence technology, and in particular relates to a model optimization method, device and storage medium based on visual semantic data enhancement. Background Art

[0002] With the rapid development of embodied intelligence technology, there is a growing demand for intelligent agents to be used in embodied tasks such as visual semantic navigation and embodied command execution. In these tasks, models need to understand the semantic information of objects in the scene, including their attributes, spatial relationships, and functional semantics. The degree of this understanding directly impacts the performance of the task. However, traditional data augmentation methods primarily expand the data size through geometric transformations (rotation, cropping), color space object material adjustments, or noise addition. While these methods can improve the model's robustness to low-level visual features, they lack explicit modeling of the semantic relevance of scene objects and fail to target defects that cause the model to fail in task execution, resulting in an insufficient match between the augmented data and the model's weaknesses. Summary of the Invention

[0003] The present invention aims to solve one of the technical problems existing in the existing related technologies to at least a certain extent.

[0004] The present invention aims to help enhance the performance of models under embodied tasks. To this end, the present invention proposes a model optimization method, device and storage medium based on visual semantic data enhancement. By utilizing the understanding ability of a large language model, the present invention extracts key object semantic information based on the performance of the model under embodied tasks, and generates new training data based on this to optimize the model.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The first aspect of the present invention provides a model optimization method based on visual semantic data enhancement, comprising:

[0007] Quantitatively evaluate the performance of the model to be optimized in the visual semantic task based on a preset data set to obtain performance data of the model to be optimized, and select the performance data corresponding to the task execution failure and the corresponding task scenario as the data to be analyzed;

[0008] Converting the data to be analyzed into a performance description of the model to be optimized and a task scenario description, and then inputting the two into a large language model to generate key semantic information associated with the defects of the model to be optimized;

[0009] Generate a new task scenario description based on the key semantic information using a large language model, construct a new three-dimensional simulation task scenario based on the new task scenario description, and generate new training data containing the key semantic information;

[0010] The model to be optimized is updated based on the training data.

[0011] In some embodiments, the quantitative evaluation of the performance of the to-be-optimized model in the visual semantic task based on a preset data set specifically includes:

[0012] Assume that the preset data set is D, and each data in the preset data set D corresponds to a visual semantic task task i , a visual semantic task i It consists of task objectives, initial poses and task scenarios;

[0013] Assume the model to be optimized is f θ , θ is the parameter of the model to be optimized, and the model to be optimized f θ It is mounted on an agent, and the action space of the agent is set to make the agent perform the visual semantic task in the preset data set D to obtain the model to be optimized f θ During the execution of the visual semantic task by the agent, for each time step t, the task goal and the observation information received by the agent are input into the model to be optimized f θ In the example, the model to be optimized f θ Perform decision reasoning, select the next action of the agent from the action space, and record each visual task i Success i ∈{True,False} and the pose of the agent at each time step during the task execution As the model to be optimized f θ For visual semantic tasks i Performance data, when Success i = True, indicating visual semantic task task i Success i = False, indicating visual semantic task i Execution failed. For the agent to perform visual semantic tasks i The pose at the tth time step in the process, where T is the total time step.

[0014] In some embodiments, the performance description of the model to be optimized is based on the pose Pose in the data to be analyzed. j The performance description of the model to be optimized is converted into a list of position descriptions of the agent. Each position description of the agent It consists of the current position attributes of the agent and the description of the position relationship between the current agent and different objects;

[0015] The task scene description is obtained by converting the task scene, and the task scene description OD j is a list of object position descriptions in the task scene at each time step, Description of the task scenario for time step t It consists of the current position attributes of each object and the description of the position relationship between each object and other different objects.

[0016] In some embodiments, the agent and an object with a positional relationship with it are respectively used as the subject and the associated object to form a main perspective relationship pair. The task scene at time step t The total amount of L t The main perspective relationship pair, then Specifically defined as follows:

[0017]

[0018] Among them, AgentName represents the name of the agent; Attribute represents the current location attribute of the agent; ObjectName l Indicates the name of the associated object in the lth main perspective relationship, Rela l Indicates the position relationship type of the lth main perspective relationship pair;

[0019] The task scenario at time step t The object numbered m and another object with a positional relationship with it are respectively regarded as the subject and the associated object and constitute a global relationship pair. The task scene at time step t There are m_K global relationship pairs with object numbered m as the main body, then the scene description The position description of the object numbered m is specifically defined as follows:

[0020] ObjectName m ,Attribute m |Rela m_1 ,ObjectName m_1 |…|Rela m_k ,ObjectNmame m_k |

[0021] …|Rela m_K ,ObjectName m_K

[0022] Among them, ObjectName m Representing a task scenario The name of the object numbered m; Attribute m Indicates the current position property of the object numbered m; ObjectName m_k Representing a task scenario The name of the associated object in the kth global relation pair; m_k Representing a task scenario The position relationship type of the k-th global relationship pair in .

[0023] In some embodiments, the position attribute is the global position of the agent or the object in the task scene, and the value is "center" or "edge";

[0024] The types of positional relationships include:

[0025] Distance relationship, used to indicate the distance between the subject and the associated object, with the value being "near" or "far";

[0026] Positional relationship, used to indicate the positional relationship between the subject and the associated object, with values including "in front", "beside", "above" or "on the surface";

[0027] Alignment relationship, used to indicate the coordinate alignment relationship between the subject and the associated object, with values including "center alignment";

[0028] The orientation relationship is used to indicate the orientation relationship between the subject and the associated object. The values include "facing" or "backwards".

[0029] In some embodiments, the key semantic information is an object position description of key objects in the task scene obtained by analyzing the large language model that affects the next action of the decision-making agent of the model to be optimized, and the key semantic information is a subset of the scene description.

[0030] In some embodiments, generating a new task scenario description based on the key semantic information using a large language model, constructing a new three-dimensional simulation task scenario based on the new task scenario description, and generating new training data containing the key semantic information specifically include:

[0031] Assume that the task scenario S for the jth piece of data to be analyzed j The key semantic information generated is Generate supplementary scene descriptions using the large language model And get a new mission scenario description

[0032] For each new mission scenario description Construct at least one new 3D simulation task scenario and describe the new task scenario The corresponding task objective O j And described by the new task scenario A new 3D simulation task scene constructed constitutes a piece of training data, and all training data constitute the training data set D new , D new ={d1,…,d m ,…,d M}, M is the amount of training data generated for J pieces of data to be analyzed, M ≥ J.

[0033] In some embodiments, the updating method adopted when updating the model to be optimized based on the training data includes fine-tuning and retraining.

[0034] A second aspect of the present invention provides a model optimization device based on visual semantic data enhancement, comprising:

[0035] A performance evaluation module is configured to quantitatively evaluate the performance of the to-be-optimized model in the visual semantic task based on a preset data set, obtain performance data of the to-be-optimized model, and select the performance data corresponding to the task execution failure and the corresponding task scenario as the data to be analyzed;

[0036] a semantic analysis module configured to convert the data to be analyzed into a performance description of the model to be optimized and a task scenario description, and then input the descriptions into a large language model to generate key semantic information associated with defects in the model to be optimized;

[0037] a training data generation module configured to generate a new task scenario description based on the key semantic information using a large language model, construct a new three-dimensional simulation task scenario based on the new task scenario description, and generate new training data containing the key semantic information;

[0038] An updating module is configured to update the model to be optimized based on the training data.

[0039] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the model optimization method described in any embodiment of the first aspect of the present invention.

[0040] The characteristics and beneficial effects of the present invention are:

[0041] The present invention matches data enhancement with model defects, extracts key semantic information from the data based on model performance, and uses this to construct new scenarios, thereby generating new training data to help optimize the model.

[0042] This invention leverages the commonsense knowledge and text completion capabilities of large language models to iteratively identify model weaknesses and generate targeted training data. This invention can be applied to the field of embodied intelligence, overcoming the shortcomings of previous data augmentation methods in visual semantic tasks, where the data augmentation approach did not adequately match model weaknesses. It provides more targeted training data, thereby improving model performance. This can be used for training tasks such as visual semantic navigation and embodied command execution, and can be further applied in numerous application scenarios, including algorithm improvement for household robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 An overall flow chart of the model optimization method based on visual semantic data enhancement provided by the embodiment of the first aspect of the present invention;

[0044] Figure 2 for Figure 1 An example of applying the model optimization method shown in the figure to a home indoor simulation task scenario;

[0045] Figure 3 This is a schematic structural diagram of an electronic device provided in accordance with an embodiment of the third aspect of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] On the contrary, this application covers any alternatives, modifications, equivalents, and solutions made within the spirit and scope of this application as defined by the claims. Furthermore, to facilitate a better understanding of this application, certain specific details are described in detail below in the detailed description of this application. Those skilled in the art will be able to fully understand this application without these details.

[0048] The first aspect of the present invention proposes a model optimization method based on visual semantic data enhancement, which generates new training data D according to the model performance. new ={d1,…,d m ,…,d M}, to update the model, where each data d m =(O,S'), O is the task goal, S' is the generated simulation task scenario.

[0049] See also Figure 1 The first embodiment of the present invention proposes a model optimization method based on visual semantic data enhancement, comprising the following steps:

[0050] Step S1: quantitatively evaluate the performance of the model to be optimized in the visual semantic task based on a preset data set to obtain performance data of the model to be optimized, and select the performance data corresponding to the task execution failure and the corresponding task scenario as the data to be analyzed;

[0051] Step S2: converting the data to be analyzed obtained in step S1 into a performance description of the model to be optimized and a task scenario description, and then inputting the two into a large language model to generate key semantic information associated with the defects of the model to be optimized;

[0052] Step S3: Generate a new scene description using the large language model based on the key semantic information generated in step S2, construct a new three-dimensional simulation scene using the new scene description, and generate new training data containing the key semantic information based on the three-dimensional simulation scene;

[0053] Step S4: Update the model to be optimized based on the training data generated in step S3.

[0054] In some embodiments, step S1 is a model evaluation stage, which specifically includes the following steps:

[0055] Step S11: Preset data set construction

[0056] Let the preset data set be D, and divide the preset data set into performance evaluation data set D analyze And the test dataset D test , satisfying D=D analyze ∪D test ,and Among them, the performance evaluation dataset D analyze Used for performance analysis and feature research of the model to be optimized, test data set D test Used for subsequent evaluation of the generalization performance of the optimized model. Each data in the preset dataset D corresponds to a visual semantic task task i , are respectively composed of the task target O, the initial pose Pose start and task scenarios S; the task target O is determined by the task type. For example, for visual semantic navigation tasks, it is the target object type, and for embodied instruction execution tasks, it is the natural language statement of the instruction. The task scenarios S in the preset dataset D are all simulation scenarios or real physical scenarios.

[0057] Step S12: Performance evaluation of the model to be optimized

[0058] Assume the model to be optimized is f θ , θ is the parameter of the model to be optimized, and the model to be optimized f θ Mounted on an intelligent agent, set the action space of the intelligent agent, and use the performance evaluation dataset D analyze Treat the optimization model as fθ The performance in the visual semantic task is quantitatively evaluated to obtain the model to be optimized f θ During the agent's task execution, for each time step t, the agent receives observation information, i.e., the RGB image from the first perspective, and combines it with the task goal to input it into the model to be optimized f θ In, f θ Perform decision reasoning and select the next action to be performed by the agent from the action space. Regarding performance data, specifically, for the performance evaluation dataset D analyze Each visual semantic task in i , record whether each visual semantic task is successful i ∈{True,False} and the pose of the agent at each time step during the execution of the visual semantic task As the model to be optimized f θ For visual semantic tasks i Performance data, Success i = True, indicating visual semantic task task i Success i = False, indicating visual semantic task i Execution failed. For the agent to perform visual semantic tasks i The pose at the tth time step in the process, where T is the total time step.

[0059] Step S13: Constructing the data to be analyzed

[0060] The model to be optimized f obtained in step S12 θ From all the performance data, the performance data and task scenarios corresponding to the task execution failure are selected as the data to be analyzed, and there are a total of J pieces of data to be analyzed.

[0061] In one embodiment of the present invention, in step S1, a visual semantic navigation task and a ProcTHOR simulation scenario are selected, and the verification dataset and the test dataset in the ProcTHOR dataset are used as the performance evaluation dataset D in the preset dataset. analyze And the test dataset D test ; The model produced by ProcTHORRL is selected as the initial model to be optimized in this embodiment. θ , set up the simulation scene with the model to be optimized f θ The action space of the agent is set to include: forward, turn right 30 degrees, turn left 30 degrees, look up 30 degrees, look down 30 degrees and stop.

[0062] In some embodiments, step S2 is a data analysis phase; the specific steps are as follows:

[0063] Step S21: Data conversion

[0064] For each piece of data to be analyzed obtained in step S1, the performance data and the task scenario are combined to convert it into a model to be optimized f θ Performance description and task scenario description, specifically,

[0065] For each performance data in the data to be analyzed, the pose Pose j Convert to the model to be optimized f θ Description of PD j , which is a list of descriptions of the intelligent body pose (that is, the pose of the model to be optimized) Each position description of the agent It consists of the current position attribute of the agent and the description of the position relationship between the current agent and different objects. The agent and an object with a position relationship with it are respectively regarded as the subject and the associated object to form a main perspective relationship pair. The task scene at time step t The total amount of L t The main perspective relationship pair, then Specifically defined as follows:

[0066]

[0067] Among them, AgentName represents the name of the agent; Attribute represents the location attribute of the agent, and its value is "center" or "edge", indicating that the agent is currently located in the global position of the task scene; l Indicates the name of the associated object in the lth main perspective relationship, Rela l Indicates the position relationship type of the lth main perspective relationship pair, and the agent is the relationship type Rela l The subject, ObjectName lThe object represented is the associated object in the relationship type. It should be noted that there may be multiple relationship types between an agent and an object, and a relationship pair only contains one relationship type; the relationship types include: 1) Distance relationship (Distance), which is used to indicate the distance between the subject and the associated object, and the value is "near" or "far"; 2) Position relationship (Position), which is used to indicate the position relationship between the subject and the associated object, and the values include "in front" (indicating that the associated object is in front of the agent), "beside" (indicating that the associated object is next to the agent), "above" (indicating that the associated object is above the agent) or "on the surface" (indicating that the associated object is on the surface of the agent); 3) Alignment relationship (Alignment), which is used to indicate the coordinate alignment relationship between the subject and the associated object, and the values include "center alignment"; 4) Rotation relationship (Rotation), which is used to indicate the orientation relationship between the subject and the associated object, and the values include "facing" or "backwards". In this embodiment, AgentName is named "Agent"; ObjectName l Object types are combined with object numbers. Object numbers are assigned sequentially based on the number of objects of the same type in the scene. For example, if there are two Mug-type objects in a room, they are named Mug-1 and Mug-2. An example of a location description is: "Agent, Center | Near, Desk-1 | In front, Bed-1."

[0068] Similarly, for the data to be analyzed with the number j, its task scenario S j Convert to scene description The scene description is a list of object position descriptions, indicating that S j The position description of all objects in the task scene description for time step t Let the position of object numbered m describe the task scenario of the object at time step t The position attributes in the object and the position relationship description between the object and different objects are composed of the task scene The object numbered m and another object with a positional relationship with it are respectively regarded as the subject and the associated object and constitute a global relationship pair. There are m_K global relationship pairs with object numbered m as the main body, then the scene description The position description of the object numbered m is specifically defined as follows:

[0069] ObjectName m ,Attribute m |Rela m_1 ,ObjectName m_1 |…|Relam_k ,ObjectName m_k |

[0070] …|Rela m_K ,ObjectName m_K

[0071] Among them, ObjectName m Indicates the name of the object numbered m in the task scene; Attribute m Indicates the current position attribute of the object numbered m, with a value of "center" or "edge", indicating that the object is currently located in the global position of the task scene; ObjectName m_k Represents the name of the associated object in the kth global relation pair in the task scene; m_k Indicates the position relationship type of the kth global relationship pair in the task scenario. The relationship type value is the same as above and will not be repeated here. For example, "Desk-1, Edge|Near, Bed-1".

[0072] Step S22: For the data to be analyzed numbered j and the corresponding model performance description PD j OD with scene description j , using a large language model combined with the corresponding task objectives O j , determine the key semantic information The key semantic information The object position description and key semantic description of the key objects that affect the next action of the agent in the optimized model decision-making task scene obtained by the large language model analysis It is a scene description OD j For example, [“DiningTable-1, Edge|Near, CounterTop-1”,”CounterTop-1, Edge|Near, DiningTable-0”] means that the large language model considers DiningTable-1 and CounterTop-1 to be key objects that influence the decision of the model to be optimized in the agent.

[0073] In some embodiments, step S3 is a data generation stage; the specific steps are as follows:

[0074] Step S31: Task scenario S for the jth piece of data to be analyzed j Generated key semantic information Generate supplementary scene descriptions using large language models And get a new mission scenario description For example, Scene description supplemented by large language models For: [“Plate-1, edge|on surface, DiningTable-1”, “Mug-1, edge|on surface, CounterTop-1”], then For example, [“DiningTable-1, edge|near,CounterTop-1”, “CounterTop-1, edge|near,DiningTable-1”, “Plate-1, edge|on the surface,DiningTable-1”, “Mug-1, edge|on the surface,CounterTop-1”]. Generating new scene descriptions based on key semantic information can help avoid over-fitting due to excessive focus on key semantic information when the model is further optimized.

[0075] Step S32: Describe each new task scenario Construct at least one new 3D simulation task scenario. Specifically, select appropriate object models and placement locations in the ProcTHOR simulation scenario as new objects, grid the ProcTHOR simulation scenario, set the grid size to 0.25 meters, and use a deep search traversal algorithm to traverse each grid and the object models placed on each grid until a solution is found that satisfies the new task scenario description. Based on this, the loading file of the ProcTHOR simulation scene is obtained to obtain a new three-dimensional simulation task scene; the new task scene is described The corresponding task objective O j And described by the new task scenario A new 3D simulation task scene constructed constitutes a piece of training data; all training data constitute the training data set D new , D new ={d1,…,d m ,…,d M}, M is the number of training data generated for J pieces of data to be analyzed, M ≥ J.

[0076] See also Figure 2 The model optimization method of an embodiment of the present invention is aimed at a home indoor simulation scene, using a large language model to analyze key objects in the task scene to generate a visual display diagram of new objects. Figure 2 The left side of the figure shows the RGB image of the scene at a certain time step. In this image, different objects are represented by circles of different colors, and the positional relationship between different objects is represented by dotted arrows. The objects close to the arrow end are related objects, and the objects far from the arrow end are the main objects. Figure 2 The yellow circle in the middle right part is the key semantic information generated by the large language model after semantic analysis of the model's performance description and task scenario description, as well as the newly added objects in the new scenario description.

[0077] In some embodiments, step S4 is a model updating stage, specifically, using the new training data set D obtained in step S3 new Update the model to be optimized f θ , and get the optimized model f θ' =ψ(D new ,f θ ), ψ is the model performance update function. It should be noted that there are many model performance update functions, such as retraining and fine-tuning, which can achieve the purpose of improving model performance. θ' is the update function of the training dataset D new The network parameters of the updated model.

[0078] In some embodiments, the optimization method of the present invention further comprises:

[0079] Step S5: Repeat steps S1 to S4 until the number of iterative optimizations reaches the preset value or the model performance improvement tends to be stable, and finally test The performance of the final model is verified on

[0080] It can be understood that the embodiment of the first aspect of the present invention provides a model optimization method based on visual semantic data enhancement, which is used to improve the model performance under visual semantic tasks. This method focuses on the impact of the semantic information of objects in the scene on the model. First, a preset data set is used to evaluate the model performance. According to the model performance and scene data, a large language model is used to summarize the key semantic information, and then this semantic information is injected into the newly generated data to generate new model training data and form a cycle of continuous improvement. Compared with traditional data enhancement methods, the present invention realizes the matching of data enhancement process and model defects, and provides a training scenario that is more suitable for improving model performance in an automated form. It can be used for multiple embodied tasks such as visual semantic navigation and visual language navigation.

[0081] A second embodiment of the present invention provides a model optimization device based on visual semantic data enhancement, comprising:

[0082] A performance evaluation module is configured to quantitatively evaluate the performance of the to-be-optimized model in the visual semantic task based on a preset data set, obtain performance data of the to-be-optimized model, and select the performance data corresponding to the task execution failure and the corresponding task scenario as the data to be analyzed;

[0083] a semantic analysis module configured to convert the data to be analyzed into a performance description of the model to be optimized and a task scenario description, and then input the descriptions into a large language model to generate key semantic information associated with defects in the model to be optimized;

[0084] a training data generation module configured to generate a new task scenario description based on the key semantic information using a large language model, construct a new three-dimensional simulation task scenario based on the new task scenario description, and generate new training data containing the key semantic information;

[0085] An updating module is configured to update the model to be optimized based on the training data.

[0086] It should be noted that the aforementioned explanations of the embodiment of the model optimization method are also applicable to the model optimization device of this embodiment and will not be repeated here.

[0087] In order to implement the above embodiment, the embodiment of the present invention further proposes a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to execute the model optimization method of the above embodiment.

[0088] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present invention. It should be noted that the electronic devices in the embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 3 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0089] like Figure 3 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. Various programs and data required for the operation of the electronic device are also stored in the RAM 103. The processing device 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0090] Typically, the following devices may be connected to the I / O interface 105: an input device 106 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 109. The communication device 109 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 3The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0091] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0092] It should be noted that the computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0093] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0094] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes the model optimization method.

[0095] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, Python, and conventional procedural programming languages such as "C-" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0096] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0097] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0098] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0099] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

[0100] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0101] Those skilled in the art will understand that all or part of the steps carried out in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the developed program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0102] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0103] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A model optimization method based on visual semantic data enhancement, characterized in that: include: Quantitatively evaluate the performance of the model to be optimized in the visual semantic task based on a preset data set to obtain performance data of the model to be optimized, and select the performance data corresponding to the task execution failure and the corresponding task scenario as the data to be analyzed; Converting the data to be analyzed into a performance description of the model to be optimized and a task scenario description, and then inputting the two into a large language model to generate key semantic information associated with the defects of the model to be optimized; Generate a new task scenario description based on the key semantic information using a large language model, construct a new three-dimensional simulation task scenario based on the new task scenario description, and generate new training data containing the key semantic information; The model to be optimized is updated based on the training data.

2. The model optimization method according to claim 1, characterized in that The quantitative evaluation of the performance of the optimized model in the visual semantic task based on the preset data set specifically includes: Assume that the preset data set is D, and each data in the preset data set D corresponds to a visual semantic task task i , a visual semantic task i It consists of task objectives, initial poses and task scenarios; Assume the model to be optimized is f θ , θ is the parameter of the model to be optimized, and the model to be optimized f θ It is mounted on an agent, and the action space of the agent is set to make the agent perform the visual semantic task in the preset data set D to obtain the model to be optimized f θ During the execution of the visual semantic task by the agent, for each time step t, the task goal and the observation information received by the agent are input into the model to be optimized f θ In the example, the model to be optimized f θ Perform decision reasoning, select the next action of the agent from the action space, and record each visual task i Success i ∈{True,False} and the pose of the agent at each time step during the task execution As the model to be optimized f θ For visual semantic tasks i Performance data, when Success i = True, indicating visual semantic task task i Success i = False, indicating visual semantic task i Execution failed. For the agent to perform visual semantic tasks i The pose at the tth time step in the process, where T is the total time step.

3. The model optimization method according to claim 2, characterized in that: The performance description of the model to be optimized is the pose Pose in the data to be analyzed j The performance description of the model to be optimized is converted into a list of position descriptions of the agent. Each position description of the agent It consists of the current position attributes of the agent and the description of the position relationship between the current agent and different objects; The task scene description is obtained by converting the task scene, and the task scene description OD j is a list of object position descriptions in the task scene at each time step, Description of the task scenario for time step t It consists of the current position attributes of each object and the description of the position relationship between each object and other different objects.

4. The model optimization method according to claim 3, characterized in that: The agent and an object with a positional relationship with it are respectively regarded as the subject and the associated object and form a main perspective relationship pair. The task scene at time step t The total amount of L t The main perspective relationship pair, then Specifically defined as follows: Among them, AgentName represents the name of the agent; Attribute represents the current location attribute of the agent; ObjectName l Indicates the name of the associated object in the lth main perspective relationship, Rela l Indicates the position relationship type of the lth main perspective relationship pair; The task scenario at time step t The object numbered m and another object with a positional relationship with it are respectively regarded as the subject and the associated object and constitute a global relationship pair. The task scene at time step t There are m_K global relationship pairs with object numbered m as the main body, then the scene description The position description of the object numbered m is specifically defined as follows: ObjectName m ,Attribute m |Rela m_1 ,ObjectName m_1 |…|Rela m_k ,ObjectName m_k |…|Rela m_K ,ObjectName m_K Among them, ObjectName m Representing a task scenario The name of the object numbered m; Attribute m Indicates the current position property of the object numbered m; ObjectName m_k Representing a task scenario The name of the associated object in the kth global relation pair; m_k Representing a task scenario The position relationship type of the k-th global relationship pair in .

5. The model optimization method according to claim 3, characterized in that: The position attribute is the global position of the agent or the object in the task scene, and the value is "center" or "edge"; The types of positional relationships include: Distance relationship, used to indicate the distance between the subject and the associated object, with the value being "near" or "far"; Positional relationship, used to indicate the positional relationship between the subject and the associated object. The values include "in front of", "beside", "above" or "on the surface"; Alignment relationship, used to indicate the coordinate alignment relationship between the subject and the associated object, with values including "center alignment"; The orientation relationship is used to indicate the orientation relationship between the subject and the associated object. The values include "facing" or "backwards".

6. The model optimization method according to claim 3, characterized in that: The key semantic information is a description of the object position of the key objects that affect the next action of the decision-making agent of the model to be optimized in the task scene obtained by analyzing the large language model. The key semantic information is a subset of the scene description.

7. The model optimization method according to claim 1, characterized in that: The method of generating a new task scenario description based on the key semantic information using a large language model, constructing a new three-dimensional simulation task scenario based on the new task scenario description, and generating new training data containing the key semantic information specifically includes: Assume that the task scenario S for the jth piece of data to be analyzed j The key semantic information generated is Generate supplementary scene descriptions using the large language model And get a new mission scenario description For each new mission scenario description Construct at least one new 3D simulation task scenario and describe the new task scenario The corresponding task objective O j And described by the new task scenario A new 3D simulation task scene constructed constitutes a piece of training data, and all training data constitute the training data set D new , D new ={d1,…,d m ,…,d M }, M is the amount of training data generated for J pieces of data to be analyzed, M ≥ J.

8. The model optimization method according to any one of claims 1 to 7, characterized in that: The updating methods adopted when updating the model to be optimized based on the training data include fine-tuning and retraining.

9. A model optimization device based on visual semantic data enhancement, characterized in that: include: A performance evaluation module is configured to quantitatively evaluate the performance of the to-be-optimized model in the visual semantic task based on a preset data set, obtain performance data of the to-be-optimized model, and select the performance data corresponding to the task execution failure and the corresponding task scenario as the data to be analyzed; a semantic analysis module configured to convert the data to be analyzed into a performance description of the model to be optimized and a task scenario description, and then input the descriptions into a large language model to generate key semantic information associated with defects in the model to be optimized; a training data generation module configured to generate a new task scenario description based on the key semantic information using a large language model, construct a new three-dimensional simulation task scenario based on the new task scenario description, and generate new training data containing the key semantic information; An updating module is configured to update the model to be optimized based on the training data.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the model optimization method according to any one of claims 1 to 8.

Images