A visual model closed-loop enhancement method based on physical simulation and parameter perception

Abstract

The present application relates to a kind of visual model closed loop enhancement method based on physical simulation and parameter perception, the present application utilizes the failure mask of test set test benchmark visual model, extracts the physical feature vector of the physical parameter of failure mask, carries out clustering to the physical feature vector of failure mask, to induce the physical scene mode that benchmark visual model will fail;According to the physical scene mode obtained, targeted generation physical perception training data;Improved benchmark visual model is reinforced training using physical perception training data;Give benchmark visual model and add modulation parameter generation branch and modulation module of adaptation benchmark visual model, modulation parameter generation branch generates adaptive modulation parameter according to the physical parameter of simulation image in physical perception training data;Modulation module uses adaptive modulation parameter to modulate the image feature map of benchmark visual model, enhances model physical understanding ability, improves model effect.

Description

Technical Field

[0001] This invention relates to the field of visual model training and enhancement technology, and in particular to a closed-loop enhancement method for visual models based on physical simulation and parameter awareness. Background Technology

[0002] With the development of simulation-to-reality (Sim2Real) technology, using simulators to generate large-scale data for model training has become a new trend. Sim2Real is a method that uses virtual simulation environments to generate data to augment real-world data, and it is widely used, especially in fields such as computer vision, robot control, and autonomous driving. By supplementing real data with simulation data, the generalization ability of models can be improved, data acquisition costs can be reduced, and the problems of data scarcity or labeling difficulties in real-world scenarios can be solved.

[0003] The process of training deep learning models using simulation-to-reality simulation involves: generating a large amount of general-purpose, scene-rich image data using existing simulation platforms (such as the game engine Unreal Engine or the autonomous driving simulator CARLA); pre-training the AI ​​model on a large scale using the simulation data; and then fine-tuning the pre-trained AI model using a small amount of real data. Data augmentation is an essential step to improve the robustness of deep learning models. Traditional data augmentation mainly involves simple geometric transformations (such as flipping and rotating) or color perturbations (brightness and contrast adjustments) at the pixel level. However, when training using simulation-to-reality simulation, various parameters (such as lighting, texture, and object position) can be subjected to extreme and random perturbations in the simulation environment, forcing the AI ​​model to learn more fundamental features that are insensitive to these changes, thereby enhancing its generalization ability.

[0004] General simulation data or completely randomized perturbation simulation data do not specifically address the "shortcomings" of AI models under specific sensors and tasks. This is a "flood" approach to enhancement, inefficient and not necessarily covering the most critical failure scenarios. Simple color perturbations or lighting from game engines usually cannot accurately reproduce the subtle and complex physical optical effects (such as highlights, vignetting, diffraction, etc.) produced by specific sensors (such as line scan cameras) and active lighting systems in the real world, resulting in insufficient physical realism. Existing methods merely "feed" the model more data, but the model itself remains a "black box" that cannot explicitly understand and utilize the physical causes behind the images; the model itself has not evolved. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, the present invention provides a visual model closed-loop enhancement method based on physical simulation and parameter perception.

[0006] In a first aspect, the present invention provides a method for visual model closed-loop enhancement based on physical simulation and parameter awareness, comprising: Step 1: Use the test set to test the failure mask of the benchmark visual model, extract the physical feature vector of the physical parameters of the failure mask, and cluster the physical feature vector of the failure mask to summarize the physical scene patterns in which the benchmark visual model will fail. Step 2: Generate physical perception training data based on the physical scene patterns obtained in Step 1; Step 3: Enhance the improved baseline visual model using physical perception training data. To enable the baseline visual model to perceive physical parameters, the following improvements are made: A modulation parameter generation branch is added to the baseline visual model, which generates adaptive modulation parameters based on the physical parameters of the simulated images in the physical perception training data; a modulation module is added to the baseline visual model, which modulates the image feature map of the baseline visual model using the adaptive modulation parameters; and the decoder of the baseline visual model decodes based on the modulated image feature map.

[0007] Furthermore, step one includes: The benchmark visual model is forward-inferred on a test set formed by real data to identify all sample regions where the benchmark visual model fails to forward-infer within the test set range, and the failure mask is obtained. The physical feature vector of each failure mask is extracted using a calibrated digital twin simulation platform. The physical feature vector includes geometric features, lighting features, material features, and camera features. Cluster the physical feature vectors of all failed samples, and use the clustering to summarize the physical scene patterns of the failure of the benchmark visual model.

[0008] Furthermore, the targeted generation of physical perception training data based on the physical scene patterns obtained in step one includes: obtaining the physical parameter ranges of each physical scene pattern summarized in step one; and using the PCG engine and a calibrated simulator to generate targeted physical perception training data rich in difficult samples. The physical perception training data adopts the following triplet format: simulation image. physical parameters The true value of the label corresponding to the simulated image .

[0009] Furthermore, the modulation parameter generation branch includes two multilayer perceptrons. The first multilayer perceptron maps physical parameters into higher-dimensional hidden physical parameter features, and the second multilayer perceptron converts the hidden physical parameter features into adaptive modulation parameters, which include a scaling modulation vector and a bias modulation vector. A modulation module is disposed between the encoder and the decoder, and the dimensions of the two modulation vectors are consistent with the number of channels of the image feature map output by the encoder. The modulation module uses the modulation vectors to modulate the deepest image feature map of the encoder.

[0010] Furthermore, the modulation parameter generation branch includes a multilayer perceptron that maps physical parameters into higher-dimensional hidden physical parameter features. Several layers of encoder and decoder units are used to convert the hidden physical parameter features into adaptive modulation parameter multilayer perceptrons. The adaptive modulation parameters include scaling modulation vectors and bias modulation vectors that adapt to the image feature maps of each encoder and decoder unit layer.

[0011] Furthermore, the adaptation modulation parameter generation branch sets up several modulation modules corresponding to the encoder unit and decoder unit layers. The modulation modules are set between the encoder unit and decoder unit layers at the same level. Each modulation module modulates the image feature map output by each encoder unit using corresponding adaptive modulation parameters.

[0012] Furthermore, the modulation module receives an image feature map x from the encoder and a scaled modulation vector from the modulation parameter generation branch. and bias modulation vector ; Use scaling modulation vector and bias modulation vector Apply an affine transformation to each channel of the image feature map x, and the calculation formula is as follows: ; in: This represents element-wise multiplication; the modulation vector is scaled using an affine transformation. and bias modulation vector Adaptively adjust the activation values ​​of the image feature map.

[0013] Furthermore, when training the improved baseline visual model end-to-end on the physical perception training data, the training objective is to minimize the difference between the baseline visual model output and the corresponding label ground truth.

[0014] Secondly, the present invention provides a visual model closed-loop enhancement device based on physical simulation and parameter perception, comprising: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit storing a computer program, and the processing unit implementing the visual model closed-loop enhancement method based on physical simulation and parameter perception by running the computer program stored in the storage unit.

[0015] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed, implements the aforementioned visual model closed-loop enhancement method based on physical simulation and parameter perception.

[0016] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art: The performance of benchmark vision models degrades in challenging scenes such as low contrast and high brightness because the convolutional kernels of their convolutional layers are fixed and cannot adapt to drastic input changes. The modulation parameter generation branch and modulation module described in this invention are equivalent to a "filter programmable based on physical parameters," generating an adaptive scaling modulation vector using the input physical parameters. and bias modulation vector Using scaling modulation vector and bias modulation vector By altering the activation values ​​of image feature maps, the robustness of the improved baseline visual model to various physical edge scenarios is fundamentally enhanced. This invention uses physical parameters as explicit inputs, allowing the improved baseline visual model to directly learn the correlation between "image features" and "physical causes." In other words, the original black-box nonlinear relationships in the baseline visual model are replaced with white-box physical process constraints. When encountering combinations of physical parameters not seen in the training set, inference can be performed based on learned physical laws, improving generalization ability. Compared to traditional data augmentation, which uses augmented data to improve model performance, the method provided in this invention is essentially targeted model augmentation. It makes the model a "physically perceptive" model that can "understand" and utilize physical parameters, enhancing interpretability and efficiency. This application's closed-loop augmentation method for visual models based on physical simulation and parameter perception provides a new and scalable paradigm for building more general and reliable AI systems. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a visual model closed-loop enhancement method based on physical simulation and parameter awareness provided in an embodiment of the present invention; Figure 2 The present invention provides a flowchart for using a test set to test the failure mask of a benchmark visual model, extracting the physical feature vector of the physical parameters of the failure mask, clustering the physical feature vector of the failure mask, and summarizing the physical scene patterns in which the benchmark visual model will fail. Figure 3 An architecture diagram of an improved Unet using the method of this application is provided as an embodiment of the present invention; Figure 4 An improved Unet architecture diagram for applying the method of this application, provided as an embodiment of the present invention; Figure 5 This is a schematic diagram of a visual model closed-loop enhancement device based on physical simulation and parameter perception provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0022] Example 1 This invention aims to address the technical challenge of existing deep learning vision models when dealing with complex real-world scenes. The inability of training data to cover all physical "corner cases" leads to insufficient robustness and poor generalization ability, resulting in a sharp decline in recognition performance when faced with low contrast, complex lighting, and dynamic blur. In short, it addresses the problem of AI models lacking "physical common sense" and being "very vulnerable" to complex real-world conditions.

[0023] To address the aforementioned shortcomings, the present invention aims to provide a novel, closed-loop, physically-aware visual model enhancement method based on physical simulation and parameter awareness. This method no longer blindly generates data, but instead: 1) intelligently "diagnoses" the physical limitations of the AI; 2) generates targeted, high-fidelity physical simulation data that can "treat the symptoms"; and 3) trains a novel AI model architecture that can directly "understand" physical parameters, thereby achieving a fundamental leap in AI performance.

[0024] See Figure 1 As shown, the visual model closed-loop enhancement method based on physical simulation and parameter awareness provided in this application includes: Step 1: Use the test set to test the failure mask of the benchmark visual model, extract the physical feature vector of the physical parameters of the failure mask, and cluster the physical feature vector of the failure mask to summarize the physical scene patterns in which the benchmark visual model will fail.

[0025] In the specific implementation process, such as Figure 2 As shown, step 1 includes: performing forward inference on the benchmark visual model on a test set formed by real data, identifying all sample regions where the benchmark visual model fails to perform forward inference within the test set range, and obtaining the "failure mask"; The physical feature vector of each failure mask is extracted using a digital twin simulation platform. The physical feature vector includes geometric features, lighting features, material features, and camera features.

[0026] Cluster the physical feature vectors of all failed samples, such as using DBSCAN clustering, and automatically summarize the physical scene patterns of the baseline visual model failure using clustering, such as "low contrast mode" and "high-light occlusion mode".

[0027] Step one identifies the physical scene patterns in which the baseline visual model exhibits weaker forward reasoning capabilities. This effectively reveals the physical scene patterns that the baseline visual model did not understand.

[0028] Step Two: Generate targeted physical perception training data based on the physical scene patterns obtained in Step One. Specifically, the physical parameter ranges for each physical scene pattern summarized in Step One are obtained. Using the PCG engine and a calibrated simulator, targeted physical perception training data rich in difficult samples is generated based on these physical parameter ranges. The physical perception training data takes the form of the following triples: simulation image... physical parameters The true value of the label corresponding to the simulated image .

[0029] Step 3: Use the physical perception training data to reinforce the improved baseline visual model.

[0030] In the specific implementation process, in order to enable the benchmark visual model to perceive physical parameters, the benchmark visual model is first improved as follows: A modulation parameter generation branch adapted to the baseline vision model is added to the baseline vision model. The modulation parameter generation branch generates adaptive modulation parameters based on the physical parameters of the simulated image. The adaptive modulation parameters are used to modulate the image feature map output by the encoder. In specific implementation, the simulated image in any triplet is input into the encoder of the baseline vision model, and the physical parameters in the triplet are input into the modulation parameter generation branch.

[0031] A modulation module adapted to the baseline vision model is added. This modulation module modulates the image feature map output by the encoder using adaptive modulation parameters, and then provides the modulated image feature map to the decoder of the baseline vision model. Under the dynamic guidance of adaptive modulation parameters formed based on physical parameters, the baseline vision model modulates the image feature map to introduce invalid physical scene pattern representations into the modulated image feature map, thereby enhancing the baseline vision model's ability to physically perceive physical scene patterns.

[0032] On the physical perception training data, the improved baseline visual model is trained end-to-end, with the training objective being to minimize the difference between the output of the baseline visual model and the corresponding label ground truth.

[0033] The baseline vision model in this application is applicable to vision models that include encoding / decoding architectures, such as the Unet model.

[0034] like Figure 3 As shown, taking Unet as an example, an improved Unet model is generated by using a modulation module and the modulation parameters output by a single adaptive modulation parameter. The process of image and physical parameter feature extraction and information fusion is as follows: Simulated images of physical perception training data The encoder backbone network is fed into Unet. The encoder backbone network consists of several encoder units, each composed of several convolutional layers and downsampling layers (such as max pooling), responsible for extracting multi-level image feature maps from the simulated image. As the encoder backbone network for extracting visual features deepens, the spatial resolution of the corresponding image feature maps gradually decreases, while the number of channels gradually increases, thereby capturing features of the simulated image from low-level texture to high-level semantics.

[0035] Simulated images for setting physical perception training data Its dimensions are 512 x 512 x 3, corresponding to height, width, and channels, physical parameters. The dimension is 10. Simulation image. After feature extraction by the first encoder unit, an image feature map with dimensions of 256x256x32 is obtained; after feature extraction by the second encoder unit, an image feature map with dimensions of 128x128x64 is obtained; after feature extraction by the third encoder unit, an image feature map with dimensions of 64x64x128 is obtained; the image feature map of the third encoder unit is used as the encoder output.

[0036] The modulation parameter generation branch includes two multilayer perceptrons. The first multilayer perceptron maps physical parameters into higher-dimensional hidden physical parameter features, and the second multilayer perceptron converts the hidden physical parameter features into adaptive modulation parameters, which include scaling modulation vectors and bias modulation vectors.

[0037] The modulation parameter generation branch uses a multilayer perceptron to process 10-dimensional physical parameters. The hidden physical parameter features are mapped to a 256-dimensional (or higher) feature space. A multilayer perceptron is responsible for mapping the discrete physical parameters to a high-dimensional feature space and learning the nonlinear relationships between the physical parameters. The modulation parameter generation branch transforms the hidden physical parameter features into adaptive modulation parameters through a second multilayer perceptron. These adaptive modulation parameters contain two modulation vectors and one scaling modulation vector. It has 128 dimensions and a bias modulation vector. The dimension is 128. Scaling the modulation vector. and bias modulation vector The dimension is consistent with the number of channels in the image feature map output by the encoder.

[0038] The modulation module utilizes the modulation vector and modulation vector The modulation module modulates the image feature map output by the encoder, without changing the dimension of the image feature map, but only changing its activation value.

[0039] The modulation module does not simply concatenate features; instead, it uses a modulation vector based on physical parameters to dynamically guide the visual feature extraction process. Specifically, the modulation module receives an image feature map x from the encoder and a scaled modulation vector from the modulation parameter generation branch. and bias modulation vector .

[0040] Use scaling modulation vector and bias modulation vector Apply an affine transformation to each channel of the image feature map x, and the calculation formula is as follows: ; in: This indicates element-wise multiplication; Through affine transformation, physical parameters It can dynamically and adaptively adjust the activation values ​​of image feature maps.

[0041] The modulation module provides the modulated image feature map to the decoder of the baseline visual model. The decoder consists of several decoder units, each composed of several convolutional layers and upsampling layers. Each decoder unit doubles the spatial dimension of the image feature map and is responsible for generating semantic segmentation masks for various targets using the multi-layered image feature map. The modulated image feature map is processed and upsampled by the first decoder unit; the input of the first decoder unit and the output of the second encoder unit are combined and provided to the second decoder unit; the second decoder unit processes and upsamples the outputs of the previous layer decoder unit and the encoder unit at the same level, then combines the final result with the output of the first encoder unit and provides it to the third decoder unit. The third decoder unit obtains a semantic segmentation mask with dimensions of 512 x 512 x the number of target types.

[0042] Under the dynamic guidance of adaptive modulation parameters formed based on physical parameters, the baseline vision model modulates the image feature map to introduce invalid physical scene pattern representations into the modulated image feature map, thereby enhancing the baseline vision model's ability to physically perceive physical scene patterns.

[0043] In specific implementation, to introduce physical parameter information between different levels of encoder / decoder units and ensure that this physical parameter information is not lost during the decoding process, this application provides another form of the modulation parameter generation branch. This branch includes a multilayer perceptron that maps physical parameters to higher-dimensional hidden physical parameter features. Several multilayer perceptrons corresponding to the encoder and decoder unit layers are used to convert the hidden physical parameter features into adaptive modulation parameter multilayer perceptrons. The adaptive modulation parameters include scaling modulation vectors and bias modulation vectors adapted to the image feature maps of each encoder and decoder unit layer. Correspondingly, the adaptive modulation parameter generation branch sets up several modulation modules corresponding to the encoder and decoder unit layers. These modulation modules are located between encoder and decoder unit layers at the same level. Each modulation module modulates the image feature map output by each encoder unit using corresponding adaptive modulation parameters. Figure 4 As shown in the Unet example, the modulation module is set between the encoder unit and decoder unit of the same level in each layer. The scaling modulation vector and bias modulation vector received by the modulation module of different levels have different dimensions. The dimension of the modulation vector corresponding to each encoder unit and decoder unit level is consistent with the number of channels of the image feature map output by the encoder unit of that level.

[0044] This application utilizes a feature modulation architecture guided by physical parameters, which has the following advantages compared to traditional data augmentation training methods for benchmark visual models: The performance of benchmark vision models degrades in challenging scenes such as low contrast and high brightness because the convolutional kernels of their convolutional layers are fixed and cannot adapt to drastic input changes. The modulation parameter generation branch and modulation module described in this invention are equivalent to a "filter programmable based on physical parameters," generating an adaptive scaling modulation vector using the input physical parameters. and bias modulation vector Using scaling modulation vector and bias modulation vector By altering the activation values ​​of image feature maps, the robustness of the improved baseline visual model to various physical edge scenarios is fundamentally enhanced. This invention uses physical parameters as explicit inputs, allowing the improved baseline visual model to directly learn the correlation between "image features" and "physical causes." In other words, the original black-box nonlinear relationships in the baseline visual model are replaced with white-box physical process constraints. When encountering combinations of physical parameters not seen in the training set, inference can be performed based on learned physical laws, improving generalization ability. Compared to traditional data augmentation, which uses augmented data to improve model performance, the method provided in this invention is essentially targeted model augmentation. It makes the model a "physically perceptive" model that can "understand" and utilize physical parameters, enhancing interpretability and efficiency. This application's closed-loop augmentation method for visual models based on physical simulation and parameter perception provides a new and scalable paradigm for building more general and reliable AI systems.

[0045] Example 2 See Figure 5 As shown, this embodiment of the invention provides a visual model closed-loop enhancement system based on physical simulation and parameter awareness, comprising: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit serving as a computer-readable storage medium, which can be used to store software programs, computer-executable programs, and modules, such as the software program, computer-executable program, and module corresponding to the visual model closed-loop enhancement method based on physical simulation and parameter awareness in this embodiment of the invention. The processing unit implements the aforementioned visual model closed-loop enhancement method based on physical simulation and parameter awareness by running the software program, computer-executable program, and module stored in the storage unit, including: Step 1: Use the test set to test the failure mask of the benchmark visual model, extract the physical feature vector of the physical parameters of the failure mask, and cluster the physical feature vector of the failure mask to summarize the physical scene patterns in which the benchmark visual model will fail. Step 2: Generate physical perception training data based on the physical scene patterns obtained in Step 1; Step 3: Enhance the training of the baseline visual model using the physical perception training data. To enable the baseline visual model to perceive physical parameters, the following improvements are made: A modulation parameter generation branch adapted to the baseline visual model is added, generating adaptive modulation parameters based on the physical parameters of the simulated images in the physical perception training data; a modulation module adapted to the baseline visual model is added, modulating the image feature map output by the encoder of the baseline visual model using the adaptive modulation parameters; the decoder of the baseline visual model decodes based on the modulated image feature map.

[0046] Of course, the computer program stored in the storage unit of the visual model closed-loop enhancement system based on physical simulation and parameter perception provided in the embodiments of the present invention is not limited to the method operation described above, but can also execute related operations in the visual model closed-loop enhancement method based on physical simulation and parameter perception provided in any embodiment of the present invention.

[0047] Example 3 This invention provides a computer-readable storage medium storing a computer program. When executed, the computer program implements the aforementioned visual model closed-loop enhancement method based on physical simulation and parameter awareness, comprising: Step 1: Use the test set to test the failure mask of the benchmark visual model, extract the physical feature vector of the physical parameters of the failure mask, and cluster the physical feature vector of the failure mask to summarize the physical scene patterns in which the benchmark visual model will fail. Step 2: Generate physical perception training data based on the physical scene patterns obtained in Step 1; Step 3: Enhance the training of the baseline visual model using the physical perception training data. To enable the baseline visual model to perceive physical parameters, the following improvements are made: A modulation parameter generation branch adapted to the baseline visual model is added, generating adaptive modulation parameters based on the physical parameters of the simulated images in the physical perception training data; a modulation module adapted to the baseline visual model is added, modulating the image feature map output by the encoder of the baseline visual model using the adaptive modulation parameters; the decoder of the baseline visual model decodes based on the modulated image feature map.

[0048] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program that is not limited to the method operation described above, and can also execute related operations in the visual model closed-loop enhancement method based on physical simulation and parameter perception provided in any embodiment of the present invention.

[0049] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0050] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0051] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0052] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for enhancing the closed-loop visual model based on physical simulation and parameter awareness, characterized in that, include: Step 1: Using a test set, test the failure mask of the benchmark visual model, extract the physical feature vectors of the physical parameters of the failure mask, and cluster the physical feature vectors of the failure mask to summarize the physical scene patterns in which the benchmark visual model will fail. Specifically, perform forward inference on the test set formed by real data to identify all sample regions where the benchmark visual model fails during forward inference within the test set, thus obtaining the failure mask. Use a calibrated digital twin simulation platform to extract the physical feature vector of each failure mask. The physical feature vector includes geometric features, lighting features, material features, and camera features. Cluster the physical feature vectors of all failure samples and use the clustering to summarize the physical scene patterns in which the benchmark visual model fails. Step 2: Generate physical perception training data based on the physical scene patterns obtained in Step 1; Step 3: Enhance the improved baseline visual model using physical perception training data. To enable the baseline visual model to perceive physical parameters, the following improvements are made: A modulation parameter generation branch is added to the baseline visual model, which generates adaptive modulation parameters based on the physical parameters of the simulated images in the physical perception training data; a modulation module is added to the baseline visual model, which modulates the image feature map of the baseline visual model using the adaptive modulation parameters; and the decoder of the baseline visual model decodes based on the modulated image feature map.

2. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, The targeted generation of physical perception training data based on the physical scene patterns obtained in Step 1 includes: obtaining the physical parameter ranges of each physical scene pattern summarized in Step 1; and using the PCG engine and a calibrated simulator to generate targeted physical perception training data rich in difficult samples. The physical perception training data adopts the following triplet format: simulation image. physical parameters The true value of the label corresponding to the simulated image .

3. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, The modulation parameter generation branch includes two multilayer perceptrons. The first multilayer perceptron maps physical parameters into higher-dimensional hidden physical parameter features. The second multilayer perceptron converts the hidden physical parameter features into adaptive modulation parameters, which include a scaling modulation vector and a bias modulation vector. A modulation module is disposed between the encoder and the decoder, and the dimensions of the two modulation vectors are consistent with the number of channels of the image feature map output by the encoder. The modulation module uses the modulation vectors to modulate the deepest image feature map of the encoder.

4. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, The modulation parameter generation branch includes a multilayer perceptron that maps physical parameters into higher-dimensional hidden physical parameter features. Several encoder and decoder units are used to convert the hidden physical parameter features into adaptive modulation parameter multilayer perceptrons. The adaptive modulation parameters include scaling modulation vectors and bias modulation vectors that adapt to the image feature maps of each encoder and decoder unit. The dimension of the modulation vector corresponding to each encoder and decoder unit level is consistent with the number of channels in the image feature map output by the encoder unit at that level.

5. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 4, characterized in that, The modulation parameter generation branch sets up several modulation modules corresponding to the encoder unit and decoder unit layers. The modulation modules are set between the encoder unit and decoder unit layers at the same level. Each modulation module modulates the image feature map output by each encoder unit using corresponding adaptive modulation parameters.

6. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 3 or 5, characterized in that, The modulation module receives the image feature map x from the encoder and the scaled modulation vector from the modulation parameter generation branch. and bias modulation vector ; Use scaling modulation vector and bias modulation vector Apply an affine transformation to each channel of the image feature map x, and the calculation formula is as follows: ； in: This represents element-wise multiplication; the modulation vector is scaled using an affine transformation. and bias modulation vector Adaptively adjust the activation values ​​of the image feature map.

7. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, When training the improved baseline visual model end-to-end on the physical perception training data, the training objective is to minimize the difference between the baseline visual model output and the corresponding label ground truth.

8. A visual model closed-loop enhancement device based on physical simulation and parameter perception, comprising: At least one processing unit is connected to a storage unit via a bus unit, characterized in that the storage unit stores a computer program, and the processing unit implements the visual model closed-loop enhancement method based on physical simulation and parameter perception as described in any one of claims 1-7 by running the computer program stored in the storage unit.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the visual model closed-loop enhancement method based on physical simulation and parameter perception as described in any one of claims 1-7.

Images