Virtual environment perception model self-evolution method, device, equipment, medium and product
By constructing a difference detection and automatic annotation between the perception stream and the benchmark truth stream in a virtual environment, the real-time self-correction of the perception model and the dynamic correction of the decision are realized. This solves the problems of insufficient generalization ability of the perception model in the virtual environment and high cost of manual intervention, and improves the accuracy and efficiency of recognition and decision-making.
Patent Information
- Application Number
- CN202610499011.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-15
- Publication Date
- 2026-07-24
Smart Images
Figure CN122452813A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent interaction technology in the metaverse, and in particular to a method, apparatus, device, medium and product for the self-evolution of a virtual environment perception model. Background Technology
[0002] In complex virtual environments such as metaverse scenarios, highly interactive and logically consistent virtual simulation environments are constructed through scenario dynamic generation and environmental cognition technologies, enabling intelligent agents within these environments to understand complex scenes and autonomously evolve. Intelligent agents need to possess the ability to accurately perceive dynamically changing scenes and continuously self-optimize to ensure the realism of interaction and the coherence of logic; this is the core technological requirement for achieving high-level intelligent interaction.
[0003] To meet these needs, existing technologies for environmental cognition typically employ a vision-based approach. Specifically, the agent primarily acquires real-time image data of the environment, such as RGB images, through its virtual camera. This image data is then input into a pre-trained visual perception model for processing. This model utilizes its learned feature extraction capabilities to perform object detection or scene segmentation, thereby identifying information such as the type, quantity, and spatial location of objects within the scene.
[0004] However, the aforementioned existing technical solutions still have the following significant shortcomings in practical applications. First, visual perception models are usually pre-trained on fixed datasets, resulting in limited generalization ability. When the virtual environment undergoes dynamic changes, such as sudden changes in lighting, unexpected damage to objects, or occlusion, the recognition accuracy of the perception model will significantly decrease. Second, existing perception systems typically operate in an open-loop manner, meaning the system is only responsible for outputting perception results but lacks a mechanism to verify these results. This leads to errors generated at the perception level persisting and accumulating at higher levels, causing cognitive biases and decision-making failures. Finally, when the performance of the perception model degrades, traditional optimization methods rely on manual intervention for data labeling and offline retraining. This process not only consumes a significant amount of manpower and time but also prevents the agent from achieving real-time, closed-loop self-evolution of its capabilities during operation. Summary of the Invention
[0005] This application provides a method, apparatus, device, medium, and product for the self-evolution of a virtual environment perception model, aiming to solve the problems of poor generalization ability, lack of verification mechanism for perception results, high cost and low efficiency caused by manual intervention in the model optimization process, and inability to achieve real-time, closed-loop self-evolution in existing technologies.
[0006] Firstly, this application provides a method for the self-evolution of a virtual environment perception model, including: A perception stream is generated by processing the perception data of the virtual environment using a first perception model; the first perception model is trained using sample data composed of historical perception data and target detection results corresponding to the historical perception data. Obtain the baseline truth stream of the virtual environment from its underlying layer; Determine the perceptual difference between the perceived stream and the reference truth stream; When the perceived difference meets the model update triggering condition, the perceptual data corresponding to the perceptual difference in the perceptual stream is automatically labeled using the benchmark truth stream to generate incremental training data. The first perception model is updated online based on the incremental training data to obtain the second perception model.
[0007] In one embodiment, after acquiring the sensing stream, the method further includes: The perception stream is input into an attribute mapping network to convert the perception features of the perception stream into text attribute labels, which represent the state of the target in the perception stream. Structured prompt words are constructed based on the category and spatial relationship of the target in the perception stream and the text attribute tags; If the perceived difference does not meet the model update triggering condition, the structured prompt words are input into the decision model for reasoning to obtain the decision result.
[0008] In one embodiment, the decision model is connected to a static domain knowledge base and a dynamic scenario state base, respectively. The static domain knowledge base is used to retrieve decision rules, and the dynamic scenario state base is used to retrieve real-time scenario states. The decision model is used to combine the decision rules, the real-time scenario states, and the structured prompts to generate the decision result.
[0009] In one embodiment, after obtaining the decision result, the method further includes: The dynamic scenario state library is updated based on the actual scenario state after the decision is executed.
[0010] In one embodiment, after obtaining the second perception model, the method further includes: Evaluate the second perceptual model on the historical validation set and on the incremental training data, using a first performance metric and a second performance metric. When the first performance metric is not lower than the third performance metric of the first perception model on the historical validation set, and the second performance metric is better than the fourth performance metric of the first perception model on the incremental training data, the model weights of the second perception model are used to replace the model weights of the first perception model without interrupting the operation of the virtual environment.
[0011] In one embodiment, the self-evolution of the virtual environment awareness model further includes: When new decision logic is generated during the interaction with the virtual environment, the new decision logic is reverse-converted into the static domain knowledge base.
[0012] Secondly, this application also provides a virtual environment perception model self-evolution device, comprising: The perception stream acquisition module is used to acquire the perception stream generated by the first perception model through processing the perception data of the virtual environment; the first perception model is obtained by training using sample data composed of historical perception data and the target detection results corresponding to the historical perception data; The truth stream acquisition module is used to acquire the baseline truth stream of the virtual environment from the underlying layer of the virtual environment; A perception difference determination module is used to determine the perception difference between the perception stream and the reference truth stream; The training data generation module is used to automatically label the perceptual data corresponding to the perceptual difference in the perceptual stream using the benchmark truth stream when the perceptual difference meets the model update triggering condition, and generate incremental training data. The model update module is used to update the first perception model online based on the incremental training data.
[0013] Thirdly, this application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned virtual environment perception model self-evolution methods.
[0014] Fourthly, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described virtual environment perception model self-evolution methods.
[0015] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described virtual environment perception model self-evolution methods. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is one of the flowcharts illustrating the self-evolution method for the virtual environment perception model provided in this application; Figure 2 This is a schematic diagram of the structure of an embodiment of an intelligent agent in a virtual environment provided in this application; Figure 3 This is the second flowchart of the self-evolution method for the virtual environment perception model provided in this application; Figure 4 This is one of the structural schematic diagrams of the virtual environment perception model self-evolution device provided in this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0020] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.
[0021] The following description, in conjunction with the accompanying drawings, describes the self-evolution method, apparatus, equipment, medium, and product for virtual environment perception models provided in this application.
[0022] It should be noted that the virtual environment perception model self-evolution method provided in this application embodiment is implemented based on the virtual environment perception model self-evolution device. The virtual environment perception model self-evolution method realizes the self-iteration of the intelligent agent's perception ability in the virtual environment, real-time environmental perception, and dynamic scenario generation by constructing a dual closed-loop system of evolution and environmental cognition.
[0023] This application describes the virtual environment perception model self-evolution method using a virtual environment perception model self-evolution device as the execution subject as an example.
[0024] Figure 1 This is one of the flowcharts illustrating the self-evolution method for the virtual environment perception model provided in this application. For example... Figure 1 As shown, the virtual environment perception model self-evolution method provided in this application includes: S110: Obtain the perception stream generated by processing the perception data of the virtual environment using the first perception model. The first perception model is trained using sample data composed of historical perception data and the target detection results corresponding to the historical perception data.
[0025] Figure 2 This diagram illustrates the structure of an embodiment of the intelligent agent in a virtual environment provided in this application. The intelligent agent includes a scene evolution engine, an environment perception toolset, a lightweight perception model, and a decision model. The scene evolution engine is used to execute the self-evolution method of the virtual environment perception model of this application.
[0026] like Figure 2 As shown, the environmental perception toolset includes visual cameras that simulate the human eye. These visual cameras are used to perceive the virtual environment and acquire perceptual data.
[0027] The perception model is used to process the perception data acquired by the vision camera to obtain a perception stream. In this application, the first perception model is the perception model before the update, and the second perception model (see subsequent description) is the perception model after the update. In one possible implementation, the perception data acquired by the vision camera is a real-time RGB image. The scene evolution engine calls the perception model to perform object detection on the real-time RGB image to obtain object detection boxes and visual feature vectors, i.e., the perception stream.
[0028] The decision model is used to reason based on the perception flow to obtain the agent's behavioral decisions, and by executing these behavioral decisions, the agent can interact with the virtual environment.
[0029] S120: Obtain the baseline truth stream of the virtual environment from the underlying layer of the virtual environment.
[0030] like Figure 2As shown, the environmental perception toolset also includes a semantic camera that simulates a God's-eye view. The semantic camera gives the agent the ability to act as an environmental probe. Specifically, the semantic camera is used to obtain the baseline truth stream of the virtual environment from its underlying layers through a ray-detection array. This includes, for example, ground truth data such as the precise coordinates, model IDs, materials, and states of objects in the virtual environment. The baseline truth stream is a type of structured metadata; in this application, it refers to the standard values of real-time data in the perception stream.
[0031] S130: Determine the perceptual difference between the perceptual stream and the reference truth stream.
[0032] Specifically, a consistency check is performed between the perceived stream and the baseline truth stream to determine perceived discrepancies. The verification metrics include, but are not limited to, the following: Existence check: Compare the difference in the number of targets (e.g., objects, people) detected in the perceived stream and the baseline truth stream.
[0033] Position verification: Calculate the intersection-union ratio (IUU) of the target detection box in the perception stream and the corresponding ray-projected bounding box in the baseline truth stream.
[0034] Semantic verification: Compare the visual recognition labels of targets in the perception stream with the corresponding entity metadata IDs in the baseline truth stream.
[0035] Each item corresponds to a perceived difference.
[0036] S140: When the perceived difference meets the model update triggering condition, the perceived data corresponding to the perceived difference in the perceived stream is automatically labeled using the benchmark truth stream to generate incremental training data.
[0037] In one possible implementation, during consistency checking, if the difference in any perceived difference exceeds a preset threshold, the current frame's perceived stream is determined to be a high-value hard example, triggering an evolutionary cycle, generating incremental training data, and updating the first perceptual model. Please refer to [reference needed]. Figure 3 .
[0038] Specifically, hard examples are automatically labeled directly using precise metadata from the benchmark truth stream to construct incremental training data. Since the labeling is based on the underlying facts of the virtual environment, the labeling accuracy is 100%, and the labeling cost is low, solving the pain points of high data labeling cost and potential errors in traditional agent training.
[0039] S150: Update the first perception model online based on the incremental training data to obtain the second perception model.
[0040] In one possible implementation, an adapter fine-tuning strategy (such as Low-Rank Adaptation (LoRA)) is employed, which updates the perception model online for incremental training data while freezing the backbone network parameters.
[0041] This application embodiment constructs a visual-semantic dual-stream verification architecture by running the perception stream of the virtual environment and the semantic detection stream based on the engine's underlying data in parallel. It uses the metadata provided by the semantic stream as truth values to monitor the inference accuracy of the perception model, thereby identifying difficult examples and solving the problem of the difficulty in automatically detecting cognitive biases in the dynamic environment of the metaverse. After identifying difficult examples, the perception model is updated online by constructing an incremental dataset, realizing real-time self-correction of the perception model and achieving a closed-loop self-evolution (i.e., evolutionary loop) of "running, correcting errors, and evolving". This allows the recognition ability of the agent in the dynamically changing virtual environment to continuously evolve over time. The generated scenario can be adaptively corrected in milliseconds based on real-time physical feedback (such as the real-time position of entities in the virtual environment and event triggering), while avoiding the continuous accumulation of errors generated at the perception level to the upper layers. In addition, this application embodiment uses the underlying data of the virtual environment to achieve automatic labeling of difficult examples, significantly reducing the human cost and time of data labeling, and solving the problem of rigid settings caused by reliance on manual presets in existing solutions.
[0042] In one possible implementation, such as Figure 3 As shown, after obtaining the second perceptual model (S150), the process also includes hot replacement of model weights, specifically including: Evaluate the second perceptual model on the historical validation set and on the incremental training data, using a first performance metric and a second performance metric. When the first performance metric is not lower than the third performance metric of the first perception model on the historical validation set, and the second performance metric is better than the fourth performance metric of the first perception model on the incremental training data, the model weights of the second perception model are used to replace the model weights of the first perception model without interrupting the operation of the virtual environment.
[0043] In this embodiment, the model weights are hot-replaced only when the verification performance of the updated perception model is improved, without interrupting the operation of the virtual environment, so as to ensure the continuity of the agent's environmental cognition and prevent catastrophic forgetting.
[0044] The implementation process of an existing scenario generation scheme is as follows: First, the entities and tasks expected to be included in the scenario are modeled, and their attribute information is converted into vectorized data and stored in a pre-set vector database. The system responds to user configuration operations, using a Gantt chart as an interaction medium to map entities and tasks to elements on different coordinate axes. The user edits the start and end times, duration, and dependencies between tasks. Finally, the Gantt chart generated in response to user configuration operations is input into a pre-set neural network model, which outputs the final scenario scheme. This scenario generation process is highly dependent on user interaction and configuration, essentially a "semi-automatic" one-way generation logic. Due to its lack of real-time perception of the virtual operating environment, the generated scenario scheme cannot be spontaneously and dynamically corrected based on real-time situational changes after the system starts running.
[0045] Based on the above considerations, in one possible implementation, such as Figure 3 As shown, after acquiring the sensing stream (S110), the process further includes: The perception stream is input into an attribute mapping network to convert the perception features of the perception stream into text attribute labels, which represent the state of the target in the perception stream. Structured prompt words are constructed based on the category and spatial relationship of the target in the perception stream and the text attribute tags; If the perceived difference does not meet the model update triggering condition, the structured prompt words are input into the decision model for reasoning to obtain the decision result.
[0046] Attribute mapping networks are used to transform visual feature vectors in a perceptual stream into textual attribute labels (such as the material state, physical damage level, and smoke state of an object in the perceptual stream) that can be processed by a decision model. In one possible implementation, the attribute mapping network can be a multilayer perceptron (MLP).
[0047] After the scene evolution engine calls the attribute mapping network to obtain text attribute labels, it integrates the target category, spatial relationship and text attribute labels in the perception stream to construct structured cue words, and calls the decision model to perform reasoning and decision-making based on the structured cue words.
[0048] In one possible implementation, the decision model can be a large language model, which can be trained using data related to the virtual environment.
[0049] This application embodiment connects an attribute mapping network as an attribute classification head in parallel to the backend of the perception model to transform continuous visual features into structured text attributes that the decision model can directly process. This solves the interface compatibility problem of pure text large language models being unable to directly understand visual vectors and the semantic alignment problem in cross-modal data transmission, significantly improving the agent's understanding of environmental details. On this basis, by integrating the visual information of the perception stream into the prompts of the decision model, the decision model has the ability to perceive the virtual environment in real time. The decisions generated by the decision model can be spontaneously and dynamically corrected according to the real-time situational changes of the virtual environment. In addition, this application embodiment adopts a collaborative architecture of lightweight perception model for environmental perception and large model for decision-making, avoiding the high computing power cost of directly using large models to process video streams.
[0050] Existing scenario generation schemes based on large language models often mix domain knowledge with real-time states. When dealing with complex and long-term metaverse deduction scenarios, large models are prone to long-term scenario state forgetting due to context window limitations. At the same time, due to the lack of a structured knowledge base independent of the large model, the model often suffers from illusions when dealing with domain-specific details, resulting in a disconnect between the generated logic and physical laws.
[0051] Based on the above considerations, in one possible implementation, the decision model is connected to a static domain knowledge base and a dynamic scenario state base, respectively. The static domain knowledge base is used to retrieve decision rules, and the dynamic scenario state base is used to retrieve real-time scenario states. The decision model is used to combine the decision rules, the real-time scenario states, and the structured prompts to generate the decision result.
[0052] To address the forgetting problem in long-term simulations, this application employs a decoupled memory management mechanism in its embodiments. Specifically, as follows: Figure 2 As shown, the intelligent agent also includes a static domain knowledge base and a dynamic scenario state base. The static domain knowledge base stores static professional rules, scenario presets, and decision logic accumulated through evolution. The dynamic scenario state base manages real-time scenario states and records the scenario evolution in real time. Real-time scenario states include real-time scenario running states, entity attributes, and spatial relationships. A scenario state is a high-level semantic snapshot of the virtual environment at a specific moment, constructed to support the decision model's reasoning. It is an abstracted, refined, and structured data set specifically used to describe key information related to task objectives and agent behavior. Scenario running states represent the progress stages or logical states of the entire scenario task. For example, currently in the "penetration phase"; "Target 1" has been successfully destroyed; the "emergency retreat" condition has been triggered, etc.
[0053] During the agent's initialization phase, the scenario evolution engine retrieves professional rules and scenario presets from the static domain knowledge base, generates logically consistent basic scenario plans through the decision model, and stores them in the scenario state library. The basic scenario plan is a top-level design blueprint and rule set used to guide and constrain the direction of scenario evolution and the agent's behavioral logic in the virtual environment. It defines the framework for the entire virtual environment interaction. The basic scenario plan is typically relatively static and preset, mainly including, but not limited to, task objectives, decision rules, event triggering conditions, and victory / failure conditions.
[0054] After determining the basic scenario, in each round of scene perception, the reasoning basis of the decision model includes not only structured prompts, but also decision rules obtained from the static domain knowledge base and real-time scenario states obtained from the dynamic scenario state library. Decision rules are used to avoid model illusions, and real-time scenario states are used to obtain contextual information, thereby obtaining decision results that conform to physical laws and professional knowledge.
[0055] This application embodiment stores static domain knowledge and real-time scenario states separately, enabling the physical layer scene state to be automatically corrected through real-time scenario states, and ensuring the cognitive accuracy of the cognitive layer through domain expertise. This effectively suppresses the state forgetting and illusion problems caused by context window limitations in long-term deduction of large language models, and significantly improves the logical consistency and realism of scenarios over long periods of time.
[0056] In one possible implementation, please refer to Figure 3 After obtaining the decision results, it also includes: The dynamic scenario state library is updated based on the actual scenario state after the decision is executed.
[0057] Specifically, the scene evolution engine receives the actual scene state output by the perception model, determines the real-time scenario state based on the actual scene state, and writes the real-time scenario state into the dynamic scenario state library to achieve overwriting and realize the closed-loop feedback of the cognitive layer (i.e., the cognitive loop), and then continues to run.
[0058] This application embodiment constructs a fully automatic self-evolving architecture of "perception-decision-update" without human intervention by updating the real-time scenario state in the dynamic scenario state library in real time, thereby improving the efficiency and accuracy of scenario generation.
[0059] In one possible implementation, please refer to Figure 3 The self-evolution method for virtual environment perception models also includes: When new decision logic is generated during the interaction with the virtual environment, the new decision logic is reverse-converted into the static domain knowledge base.
[0060] Specifically, the scenario evolution engine automatically identifies the decision logic corresponding to the decisions generated during the interaction process, thereby identifying new decision logic. This new decision logic is then input into the static domain knowledge base, thus updating and enhancing the static domain knowledge base.
[0061] This application's embodiments reverse-engineer the new decision-making logic generated during the interaction between the intelligent agent and the virtual environment, and deposit it into a static domain knowledge base, thereby achieving continuous accumulation of cognitive layer decision-making logic. Simultaneously, by overwriting the dynamic scenario state library when the environment changes, real-time updates of the physical layer environment state are achieved, thus constructing a dual self-evolutionary loop of cognitive and physical states to ensure the accuracy of the final decision.
[0062] Based on the above, this application also provides a virtual environment perception model self-evolution device. The virtual environment perception model self-evolution device and the aforementioned virtual environment perception model self-evolution method can be referred to and correspond to each other.
[0063] As an example, such as Figure 4 As shown, the virtual environment perception model self-evolution device provided in this application includes: The perception stream acquisition module 410 is used to acquire the perception stream generated by the first perception model through processing the perception data of the virtual environment; the first perception model is obtained by training using sample data composed of historical perception data and the target detection results corresponding to the historical perception data. The truth stream acquisition module 420 is used to acquire the baseline truth stream of the virtual environment from the underlying layer of the virtual environment; The perception difference determination module 430 is used to determine the perception difference between the perception stream and the reference truth stream; The training data generation module 440 is used to automatically label the perceptual data corresponding to the perceptual difference in the perceptual stream using the benchmark truth stream when the perceptual difference meets the model update triggering condition, and generate incremental training data. The model update module 450 is used to update the first perception model online based on the incremental training data.
[0064] This application embodiment constructs a visual-semantic dual-stream verification architecture by running the perception stream of the virtual environment and the semantic detection stream based on the engine's underlying data in parallel. It uses the metadata provided by the semantic stream as truth values to monitor the inference accuracy of the perception model, thereby identifying difficult examples and solving the problem of the difficulty in automatically detecting cognitive biases in the dynamic environment of the metaverse. After identifying difficult examples, the perception model is updated online by constructing an incremental dataset, realizing real-time self-correction of the perception model and achieving a closed-loop self-evolution of "running, correcting errors, and evolving." This allows the agent's recognition ability in the dynamically changing virtual environment to continuously evolve over time, while avoiding the continuous accumulation of errors generated at the perception level to the upper layers. In addition, this application embodiment uses the underlying data of the virtual environment to automatically label difficult examples, significantly reducing the human cost and time of data labeling, while solving the problem of rigid settings caused by reliance on manual presets in existing solutions.
[0065] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a virtual environment awareness model self-evolution method, which includes: A perception stream is generated by processing the perception data of the virtual environment using a first perception model; the first perception model is trained using sample data composed of historical perception data and target detection results corresponding to the historical perception data. Obtain the baseline truth stream of the virtual environment from its underlying layer; Determine the perceptual difference between the perceived stream and the reference truth stream; When the perceived difference meets the model update triggering condition, the perceptual data corresponding to the perceptual difference in the perceptual stream is automatically labeled using the benchmark truth stream to generate incremental training data. The first perception model is updated online based on the incremental training data to obtain the second perception model.
[0066] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0067] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the virtual environment perception model self-evolution method provided in the above embodiments. The method includes: A perception stream is generated by processing the perception data of the virtual environment using a first perception model; the first perception model is trained using sample data composed of historical perception data and target detection results corresponding to the historical perception data. Obtain the baseline truth stream of the virtual environment from its underlying layer; Determine the perceptual difference between the perceived stream and the reference truth stream; When the perceived difference meets the model update triggering condition, the perceptual data corresponding to the perceptual difference in the perceptual stream is automatically labeled using the benchmark truth stream to generate incremental training data. The first perception model is updated online based on the incremental training data to obtain the second perception model.
[0068] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the virtual environment perception model self-evolution method provided in the above embodiments, the method comprising: A perception stream is generated by processing the perception data of the virtual environment using a first perception model; the first perception model is trained using sample data composed of historical perception data and target detection results corresponding to the historical perception data. Obtain the baseline truth stream of the virtual environment from its underlying layer; Determine the perceptual difference between the perceived stream and the reference truth stream; When the perceived difference meets the model update triggering condition, the perceptual data corresponding to the perceptual difference in the perceptual stream is automatically labeled using the benchmark truth stream to generate incremental training data. The first perception model is updated online based on the incremental training data to obtain the second perception model.
[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0070] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for the self-evolution of a virtual environment perception model, characterized in that, include: The first perception model is used to process the perception data of the virtual environment to generate a perception stream. The first perception model is obtained by training sample data composed of historical perception data and target detection results corresponding to the historical perception data; Obtain the baseline truth stream of the virtual environment from its underlying layer; Determine the perceptual difference between the perceived stream and the reference truth stream; When the perceived difference meets the model update triggering condition, the perceptual data corresponding to the perceptual difference in the perceptual stream is automatically labeled using the benchmark truth stream to generate incremental training data. The first perception model is updated online based on the incremental training data to obtain the second perception model.
2. The self-evolution method for virtual environment perception model according to claim 1, characterized in that, After acquiring the perception stream, the method further includes: The perception stream is input into an attribute mapping network to convert the perception features of the perception stream into text attribute labels, which represent the state of the target in the perception stream. Structured prompt words are constructed based on the category and spatial relationship of the target in the perception stream and the text attribute tags; If the perceived difference does not meet the model update triggering condition, the structured prompt words are input into the decision model for reasoning to obtain the decision result.
3. The self-evolution method for virtual environment perception model according to claim 2, characterized in that, The decision model is connected to a static domain knowledge base and a dynamic scenario state base, respectively. The static domain knowledge base is used to retrieve decision rules, and the dynamic scenario state base is used to retrieve real-time scenario states. The decision model is used to combine the decision rules, the real-time scenario states, and the structured prompts to generate the decision result.
4. The self-evolution method for virtual environment perception model according to claim 3, characterized in that, After obtaining the decision result, the process also includes: The dynamic scenario state library is updated based on the actual scenario state after the decision is executed.
5. The self-evolution method for virtual environment perception model according to claim 1, characterized in that, After obtaining the second perceptual model, it also includes: Evaluate the second perceptual model on the historical validation set and on the incremental training data, using a first performance metric and a second performance metric. When the first performance metric is not lower than the third performance metric of the first perception model on the historical validation set, and the second performance metric is better than the fourth performance metric of the first perception model on the incremental training data, the model weights of the second perception model are used to replace the model weights of the first perception model without interrupting the operation of the virtual environment.
6. The self-evolution method for virtual environment perception model according to claim 3, characterized in that, The virtual environment perception model self-evolution method also includes: When new decision logic is generated during the interaction with the virtual environment, the new decision logic is reverse-converted into the static domain knowledge base.
7. A virtual environment perception model self-evolution device, characterized in that, include: The perception stream acquisition module is used to acquire the perception stream generated by the first perception model through processing the perception data of the virtual environment; The first perception model is obtained by training sample data composed of historical perception data and target detection results corresponding to the historical perception data; The truth stream acquisition module is used to acquire the baseline truth stream of the virtual environment from the underlying layer of the virtual environment; A perception difference determination module is used to determine the perception difference between the perception stream and the reference truth stream; The training data generation module is used to automatically label the perceptual data corresponding to the perceptual difference in the perceptual stream using the benchmark truth stream when the perceptual difference meets the model update triggering condition, and generate incremental training data. The model update module is used to update the first perception model online based on the incremental training data.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the virtual environment perception model self-evolution method as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium, wherein a computer program is stored on the non-transitory computer-readable storage medium, characterized in that, When the computer program is executed by the processor, it implements the virtual environment perception model self-evolution method as described in any one of claims 1 to 6.
10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the virtual environment perception model self-evolution method as described in any one of claims 1 to 6.