Visual model closed-loop enhancement method based on physical simulation and parameter perception

Abstract

The invention relates to a visual model closed-loop enhancement method based on physical simulation and parameter perception, and the method comprises the steps: testing a failure mask of a reference visual model through a test set, extracting a physical feature vector of a physical parameter of the failure mask, carrying out the clustering of the physical feature vector of the failure mask, and carrying out the recognition of the physical feature vector of the failure mask; concluding a physical scene mode in which the reference visual model is invalid; generating physical perception training data in a targeted manner according to the obtained physical scene mode; carrying out enhanced training on the improved reference vision model by utilizing physical perception training data; adding a modulation parameter generation branch and a modulation module which are adaptive to the reference visual model to the reference visual model, wherein the modulation parameter generation branch generates adaptive modulation parameters according to physical parameters of a simulation image in the physical perception training data; and the modulation module modulates the image feature map of the reference visual model by using the adaptive modulation parameter, enhances the physical understanding ability of the model, and improves the model effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual model training enhancement, in particular to a visual model closed-loop enhancement method based on physical simulation and parameter perception. BACKGROUND

[0002] With the development of simulation-to-reality (Sim2Real) technology, using a simulator to generate large-scale data to train a model has become a new trend. Sim2Real is a method of using a virtual simulation environment to generate data to enhance real-world data, which is widely used in computer vision, robot control and autonomous driving, etc. By supplementing the deficiency of real data with simulation data, the generalization ability of the model can be improved, the data collection cost can be reduced, and the problem of data scarcity or labeling difficulty in real scenarios can be solved.

[0003] The process of using Sim2Real for deep learning model training includes: using existing simulation platforms (such as game engines Unreal Engine or autonomous driving simulator CARLA) to generate a large amount of general and scene-rich image data; using simulation data to pre-train an AI model on a large scale, and then fine-tuning the pre-trained AI model with a small amount of real data. In order to improve the robustness of the deep learning model, data enhancement is an essential step. Traditional data enhancement mainly performs simple geometric transformations (such as flipping, rotation) or color perturbations (brightness, contrast adjustment) at the pixel level. When training with Sim2Real, various parameters (such as lighting, texture, object position) can be randomly perturbed in the simulation environment to force the AI model to learn more essential features that are not sensitive to these changes, thereby enhancing its generalization ability.

[0004] General simulation data or completely randomly perturbed simulation data does not specifically address the "shortcomings" of AI models in specific sensors and specific tasks. This is a "flood irrigation" type of enhancement, which is inefficient and may not cover the most critical failure scenarios. Simple color perturbation or lighting in game engines cannot accurately reproduce the subtle and complex physical optical effects (such as highlights, dark corners, diffraction, etc.) produced by specific sensors (such as line-scan cameras) and active lighting systems in the real world, and the physical reality is insufficient. The existing method only "feeds" the model with more data, but the model itself is still a "black box" and cannot explicitly understand and utilize the physical cause information behind the image. The model itself has not evolved. SUMMARY

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a visual model closed-loop enhancement method based on physical simulation and parameter perception.

[0006] In a first aspect, the present application provides a visual model closed-loop enhancement method based on physical simulation and parameter perception, comprising: Step one: test the failure mask of the benchmark visual model using the test set, extract the physical feature vector of the physical parameter of the failure mask, and cluster the physical feature vector of the failure mask to induce the physical scene mode in which the benchmark visual model fails; Step two: target generation of physical perception training data according to the physical scene mode obtained in step one; Step three: use physical perception training data to strengthen the training of the improved benchmark visual model; in order to enable the benchmark visual model to perceive the physical parameter, the benchmark visual model is improved as follows: add a modulation parameter generation branch to the benchmark visual model, which generates adaptive modulation parameters according to the physical parameters of the simulation image in the physical perception training data; add a modulation module to the benchmark visual model, which modulates the image feature map of the benchmark visual model using the adaptive modulation parameters; the decoder of the benchmark visual model decodes based on the modulated image feature map.

[0007] Further, step one comprises: Forward inference of the benchmark visual model on the test set formed by real data, identifying all sample regions in which the benchmark visual model fails in forward inference within the test set range, and obtaining a failure mask; Extract the physical feature vector of each failure mask using the calibrated digital twin simulation platform, which contains geometric features, lighting features, material features, and camera features; Cluster the physical feature vectors of all failure samples, and use clustering to induce the physical scene mode in which the benchmark visual model fails.

[0008] Further, target generation of physical perception training data according to the physical scene mode obtained in step one comprises: obtaining the physical parameter range of each physical scene mode induced in step one; using the PCG engine and the calibrated simulator to target generate physical perception training data rich in difficult samples, which adopts the following triple format: simulation image , physical parameter , label true value corresponding to the simulation image .

[0009] Further, the modulation parameter generation branch contains two multilayer perceptrons, the first multilayer perceptron maps the physical parameter into a higher-dimensional hidden physical parameter feature, and the second multilayer perceptron converts the hidden physical parameter feature into an adaptive modulation parameter, which contains a scaling modulation vector and a bias modulation vector. One of the modulation modules is arranged between the encoder and the decoder, and the dimensions of the two modulation vectors are consistent with the number of channels of the image feature map output by the encoder, and the modulation module modulates the image feature map at the deepest layer of the encoder using the modulation vectors.

[0010] Further, the modulation parameter generation branch includes a multi-layer perception machine for mapping the physical parameters into higher-dimensional hidden physical parameter features, and a multi-layer perception machine corresponding to the number of layers of the encoder unit and the decoder unit for converting the hidden physical parameter features into adaptive modulation parameters, wherein the adaptive modulation parameters include scaling modulation vectors and bias modulation vectors adapted to the image feature maps of each layer of the encoder unit and the decoder unit.

[0011] Further, the adaptive modulation parameter generation branch is arranged with the modulation modules corresponding to the number of layers of the encoder unit and the decoder unit, and each of the modulation modules is arranged between the encoder unit and the decoder unit at the same layer, and each of the modulation modules modulates the image feature map output by the corresponding encoder unit using the corresponding adaptive modulation parameters.

[0012] Further, the modulation module receives the image feature map x from the encoder and the scaling modulation vector and the bias modulation vector ; An affine transformation is applied to each channel of the image feature map x using the scaling modulation vector and the bias modulation vector , and the calculation formula is: ; Wherein: represents element-wise multiplication; through the affine transformation, the scaling modulation vector and the bias modulation vector adaptively adjust the activation values of the image feature map.

[0013] Further, when the improved benchmark visual model is trained end-to-end on the physical perception training data, the training target is to minimize the difference between the output of the benchmark visual model and the corresponding label true value.

[0014] In a second aspect, the present application provides a visual model closed-loop enhancement device based on physical simulation and parameter perception, comprising: at least one processing unit, the processing unit is connected to the storage unit through the bus unit, the storage unit stores the computer program, and the processing unit realizes the visual model closed-loop enhancement method based on physical simulation and parameter perception by running the computer program stored in the storage unit.

[0015] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed, implements the aforementioned visual model closed-loop enhancement method based on physical simulation and parameter perception.

[0016] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art: The performance of benchmark vision models degrades in challenging scenes such as low contrast and high brightness because the convolutional kernels of their convolutional layers are fixed and cannot adapt to drastic input changes. The modulation parameter generation branch and modulation module described in this invention are equivalent to a "filter programmable based on physical parameters," generating an adaptive scaling modulation vector using the input physical parameters. and bias modulation vector Using scaling modulation vector and bias modulation vector By altering the activation values ​​of image feature maps, the robustness of the improved baseline visual model to various physical edge scenarios is fundamentally enhanced. This invention uses physical parameters as explicit inputs, allowing the improved baseline visual model to directly learn the correlation between "image features" and "physical causes." In other words, the original black-box nonlinear relationships in the baseline visual model are replaced with white-box physical process constraints. When encountering combinations of physical parameters not seen in the training set, inference can be performed based on learned physical laws, improving generalization ability. Compared to traditional data augmentation, which uses augmented data to improve model performance, the method provided in this invention is essentially targeted model augmentation. It makes the model a "physically perceptive" model that can "understand" and utilize physical parameters, enhancing interpretability and efficiency. This application's closed-loop augmentation method for visual models based on physical simulation and parameter perception provides a new and scalable paradigm for building more general and reliable AI systems. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a visual model closed-loop enhancement method based on physical simulation and parameter awareness provided in an embodiment of the present invention; Figure 2 A flowchart is provided for the embodiment of the present application, which utilizes a test set to test the failure mask of the benchmark visual model, extracts the physical feature vector of the physical parameters of the failure mask, and clusters the physical feature vector of the failure mask to induce the physical scene mode in which the benchmark visual model fails; Figure 3 An architecture diagram of an improved Unet applying the method of the present application is provided for the embodiment of the present application. Figure 4 Another architecture diagram of an improved Unet applying the method of the present application is provided for the embodiment of the present application. Figure 5 A schematic diagram of a visual model closed-loop enhancement device based on physical simulation and parameter perception is provided for the embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0021] It should be noted that in this document, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0022] Embodiment 1 The present application aims to solve the technical problems of insufficient model robustness, poor generalization ability, and sharp decline in recognition performance in the face of low contrast, complex lighting, dynamic blur, etc. when existing deep learning visual models process complex real-world scenes, because the training data cannot cover all physical "corner cases". In short, it is to solve the problem that AI models lack "physical common sense" and are "very fragile" in the face of complex reality.

[0023] In view of the above shortcomings, the purpose of the present application is to provide a brand-new, closed-loop, physically-aware visual model closed-loop enhancement method based on physical simulation and parameter perception. Instead of blindly generating data, the method intelligently "diagnoses" the physical shortcomings of AI, targets the generation of high-fidelity physical simulation data that can "treat the disease", and trains a new AI model architecture that can directly "understand" physical parameters, thereby achieving a substantial leap in AI performance.

[0024] Referring to Figure 1 The visual model closed-loop enhancement method based on physical simulation and parameter perception provided by the present application comprises: Step one: use the test set to test the failure mask of the benchmark visual model, extract the physical feature vector of the physical parameters of the failure mask, and cluster the physical feature vector of the failure mask to induce the physical scene mode in which the benchmark visual model will fail.

[0025] In the specific implementation process, as Figure 2 Step 1 includes: performing forward inference on the benchmark visual model on the test set formed by real data, identifying all sample regions in which the benchmark visual model fails in forward inference within the test set range, and obtaining a "failure mask"; Use the digital twin simulation platform to extract the physical feature vector of each failure mask, which includes geometric features, lighting features, material features, and camera features.

[0026] Cluster the physical feature vectors of all failure samples, such as DBSCAN clustering, and automatically induce the physical scene mode in which the benchmark visual model fails, such as "low contrast mode" and "high light shadowing mode".

[0027] Step one summarizes the benchmark visual model's poorer forward inference ability in which physical scene mode. In fact, the physical scene mode that the benchmark visual model does not understand is obtained.

[0028] Step two: target generation of physical perception training data according to the physical scene mode obtained in step one. In the specific implementation process, the physical parameter range of each physical scene mode induced in step one is obtained; PCG engine and calibrated simulator are used to target generation of physically-aware training data with difficult samples according to the physical parameter range, which takes the form of the following triplets: simulation image , physical parameters , and label true value corresponding to the simulation image .

[0029] Step three: use the physically-aware training data to strengthen the training of the improved benchmark visual model.

[0030] In the specific implementation process, in order to enable the benchmark visual model to perceive physical parameters, the benchmark visual model is first improved as follows: A modulation parameter generation branch adapted to the baseline vision model is added to the baseline vision model. The modulation parameter generation branch generates adaptive modulation parameters based on the physical parameters of the simulated image. The adaptive modulation parameters are used to modulate the image feature map output by the encoder. In specific implementation, the simulated image in any triplet is input into the encoder of the baseline vision model, and the physical parameters in the triplet are input into the modulation parameter generation branch.

[0031] A modulation module adapted to the baseline vision model is added. This modulation module modulates the image feature map output by the encoder using adaptive modulation parameters, and then provides the modulated image feature map to the decoder of the baseline vision model. Under the dynamic guidance of adaptive modulation parameters formed based on physical parameters, the baseline vision model modulates the image feature map to introduce invalid physical scene pattern representations into the modulated image feature map, thereby enhancing the baseline vision model's ability to physically perceive physical scene patterns.

[0032] On the physical perception training data, the improved baseline visual model is trained end-to-end, with the training objective being to minimize the difference between the output of the baseline visual model and the corresponding label ground truth.

[0033] The baseline vision model in this application is applicable to vision models that include encoding / decoding architectures, such as the Unet model.

[0034] like Figure 3 As shown, taking Unet as an example, an improved Unet model is generated by using a modulation module and the modulation parameters output by a single adaptive modulation parameter. The process of image and physical parameter feature extraction and information fusion is as follows: Simulated images of physical perception training data The encoder backbone network is fed into Unet. The encoder backbone network consists of several encoder units, each composed of several convolutional layers and downsampling layers (such as max pooling), responsible for extracting multi-level image feature maps from the simulated image. As the encoder backbone network for extracting visual features deepens, the spatial resolution of the corresponding image feature maps gradually decreases, while the number of channels gradually increases, thereby capturing features of the simulated image from low-level texture to high-level semantics.

[0035] Simulated images for setting physical perception training data Its dimensions are 512 x 512 x 3, corresponding to height, width, and channels, physical parameters. The dimension is 10. Simulation image. After feature extraction by the first encoder unit, an image feature map with dimensions of 256x256x32 is obtained; after feature extraction by the second encoder unit, an image feature map with dimensions of 128x128x64 is obtained; after feature extraction by the third encoder unit, an image feature map with dimensions of 64x64x128 is obtained; the image feature map of the third encoder unit is used as the encoder output.

[0036] The modulation parameter generation branch includes two multilayer perceptrons. The first multilayer perceptron maps physical parameters into higher-dimensional hidden physical parameter features, and the second multilayer perceptron converts the hidden physical parameter features into adaptive modulation parameters, which include scaling modulation vectors and bias modulation vectors.

[0037] The modulation parameter generation branch uses a multilayer perceptron to process 10-dimensional physical parameters. The hidden physical parameter features are mapped to a 256-dimensional (or higher) feature space. A multilayer perceptron is responsible for mapping the discrete physical parameters to a high-dimensional feature space and learning the nonlinear relationships between the physical parameters. The modulation parameter generation branch transforms the hidden physical parameter features into adaptive modulation parameters through a second multilayer perceptron. These adaptive modulation parameters contain two modulation vectors and one scaling modulation vector. It has 128 dimensions and a bias modulation vector. The dimension is 128. Scaling the modulation vector. and bias modulation vector The dimension is consistent with the number of channels in the image feature map output by the encoder.

[0038] The modulation module utilizes the modulation vector and modulation vector The modulation module modulates the image feature map output by the encoder, without changing the dimension of the image feature map, but only changing its activation value.

[0039] The modulation module does not simply concatenate features; instead, it uses a modulation vector based on physical parameters to dynamically guide the visual feature extraction process. Specifically, the modulation module receives an image feature map x from the encoder and a scaled modulation vector from the modulation parameter generation branch. and bias modulation vector .

[0040] Use scaling modulation vector and bias modulation vector Apply an affine transformation to each channel of the image feature map x, and the calculation formula is as follows: ; in: This indicates element-wise multiplication; Through affine transformation, physical parameters It can dynamically and adaptively adjust the activation values ​​of image feature maps.

[0041] The modulation module provides the modulated image feature map to the decoder of the baseline visual model. The decoder consists of several decoder units, each composed of several convolutional layers and upsampling layers. Each decoder unit doubles the spatial dimension of the image feature map and is responsible for generating semantic segmentation masks for various targets using the multi-layered image feature map. The modulated image feature map is processed and upsampled by the first decoder unit; the input of the first decoder unit and the output of the second encoder unit are combined and provided to the second decoder unit; the second decoder unit processes and upsamples the outputs of the previous layer decoder unit and the encoder unit at the same level, then combines the final result with the output of the first encoder unit and provides it to the third decoder unit. The third decoder unit obtains a semantic segmentation mask with dimensions of 512 x 512 x the number of target types.

[0042] Under the dynamic guidance of adaptive modulation parameters formed based on physical parameters, the baseline vision model modulates the image feature map to introduce invalid physical scene pattern representations into the modulated image feature map, thereby enhancing the baseline vision model's ability to physically perceive physical scene patterns.

[0043] In specific implementation, to introduce physical parameter information between different levels of encoder / decoder units and ensure that this physical parameter information is not lost during the decoding process, this application provides another form of the modulation parameter generation branch. This branch includes a multilayer perceptron that maps physical parameters to higher-dimensional hidden physical parameter features. Several multilayer perceptrons corresponding to the encoder and decoder unit layers are used to convert the hidden physical parameter features into adaptive modulation parameter multilayer perceptrons. The adaptive modulation parameters include scaling modulation vectors and bias modulation vectors adapted to the image feature maps of each encoder and decoder unit layer. Correspondingly, the adaptive modulation parameter generation branch sets up several modulation modules corresponding to the encoder and decoder unit layers. These modulation modules are located between encoder and decoder unit layers at the same level. Each modulation module modulates the image feature map output by each encoder unit using corresponding adaptive modulation parameters. Figure 4 As shown in the Unet example, the modulation module is set between the encoder unit and decoder unit of the same level in each layer. The scaling modulation vector and bias modulation vector received by the modulation module of different levels have different dimensions. The dimension of the modulation vector corresponding to each encoder unit and decoder unit level is consistent with the number of channels of the image feature map output by the encoder unit of that level.

[0044] This application utilizes a feature modulation architecture guided by physical parameters, which has the following advantages compared to traditional data augmentation training methods for benchmark visual models: The performance of benchmark vision models degrades in challenging scenes such as low contrast and high brightness because the convolutional kernels of their convolutional layers are fixed and cannot adapt to drastic input changes. The modulation parameter generation branch and modulation module described in this invention are equivalent to a "filter programmable based on physical parameters," generating an adaptive scaling modulation vector using the input physical parameters. and bias modulation vector Using scaling modulation vector and bias modulation vector By altering the activation values ​​of image feature maps, the robustness of the improved baseline visual model to various physical edge scenarios is fundamentally enhanced. This invention uses physical parameters as explicit inputs, allowing the improved baseline visual model to directly learn the correlation between "image features" and "physical causes." In other words, the original black-box nonlinear relationships in the baseline visual model are replaced with white-box physical process constraints. When encountering combinations of physical parameters not seen in the training set, inference can be performed based on learned physical laws, improving generalization ability. Compared to traditional data augmentation, which uses augmented data to improve model performance, the method provided in this invention is essentially targeted model augmentation. It makes the model a "physically perceptive" model that can "understand" and utilize physical parameters, enhancing interpretability and efficiency. This application's closed-loop augmentation method for visual models based on physical simulation and parameter perception provides a new and scalable paradigm for building more general and reliable AI systems.

[0045] Example 2 See Figure 5 As shown, this embodiment of the invention provides a visual model closed-loop enhancement system based on physical simulation and parameter awareness, comprising: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit serving as a computer-readable storage medium, which can be used to store software programs, computer-executable programs, and modules, such as the software program, computer-executable program, and module corresponding to the visual model closed-loop enhancement method based on physical simulation and parameter awareness in this embodiment of the invention. The processing unit implements the aforementioned visual model closed-loop enhancement method based on physical simulation and parameter awareness by running the software program, computer-executable program, and module stored in the storage unit, including: Step 1: Use the test set to test the failure mask of the benchmark visual model, extract the physical feature vector of the physical parameters of the failure mask, and cluster the physical feature vector of the failure mask to summarize the physical scene patterns in which the benchmark visual model will fail. Step two: target generation of physical perception training data according to the physical scene mode obtained in step one; Step three: reinforcement training of the benchmark visual model using the physical perception training data; in order to enable the benchmark visual model to perceive the physical parameters, the benchmark visual model is first improved as follows: a modulation parameter generation branch adapting to the benchmark visual model is added to the benchmark visual model, the modulation parameter generation branch generates adaptive modulation parameters according to the physical parameters of the simulation images in the physical perception training data; a modulation module adapting to the benchmark visual model is added to the benchmark visual model, the modulation module modulates the image feature map output by the encoder of the benchmark visual model using the adaptive modulation parameters; the decoder of the benchmark visual model decodes based on the modulated image feature map.

[0046] Of course, the computer program stored in the storage unit in the visual model closed-loop enhancement system based on physical simulation and parameter perception provided by the embodiment of the present application is not limited to the method operations described above, and can also perform related operations in the visual model closed-loop enhancement method based on physical simulation and parameter perception provided by any embodiment of the present application.

[0047] Embodiment 3 The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the visual model closed-loop enhancement method based on physical simulation and parameter perception, including: Step one: test the failure mask of the benchmark visual model using the test set, extract the physical feature vector of the physical parameters of the failure mask, and cluster the physical feature vector of the failure mask to induce the physical scene mode in which the benchmark visual model will fail; Step two: target generation of physical perception training data according to the physical scene mode obtained in step one; Step three: reinforcement training of the benchmark visual model using the physical perception training data; in order to enable the benchmark visual model to perceive the physical parameters, the benchmark visual model is first improved as follows: a modulation parameter generation branch adapting to the benchmark visual model is added to the benchmark visual model, the modulation parameter generation branch generates adaptive modulation parameters according to the physical parameters of the simulation images in the physical perception training data; a modulation module adapting to the benchmark visual model is added to the benchmark visual model, the modulation module modulates the image feature map output by the encoder of the benchmark visual model using the adaptive modulation parameters; the decoder of the benchmark visual model decodes based on the modulated image feature map.

[0048] The computer program stored in the computer readable storage medium provided by the embodiment of the present application is not limited to the method operation as described above, and can also perform the related operation in the visual model closed-loop enhancement method based on physical simulation and parameter perception provided by any embodiment of the present application.

[0049] In the embodiments provided by the present application, it should be understood that the disclosed structure and method can be implemented in other ways. For example, the above-described structural embodiments are merely illustrative. For example, the division of the units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interface, structure or unit, and can be electrical, mechanical or in other forms.

[0050] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0051] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0052] The above is only a specific implementation of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A method for enhancing the closed-loop visual model based on physical simulation and parameter awareness, characterized in that, include: Step 1: Use the test set to test the failure mask of the benchmark visual model, extract the physical feature vector of the physical parameters of the failure mask, and cluster the physical feature vector of the failure mask to summarize the physical scene patterns in which the benchmark visual model will fail. Step 2: Generate physical perception training data based on the physical scene patterns obtained in Step 1; Step 3: Strengthen the improved baseline vision model using physical perception training data; wherein, in order to enable the baseline vision model to perceive physical parameters, the baseline vision model is improved as follows: add a modulation parameter generation branch adapted to the baseline vision model, wherein the modulation parameter generation branch generates adaptive modulation parameters based on the physical parameters of the simulated image in the physical perception training data. A modulation module adapted to the benchmark visual model is added to the benchmark visual model. The modulation module modulates the image feature map of the benchmark visual model using adaptive modulation parameters. The decoder of the benchmark visual model decodes based on the modulated image feature map.

2. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, Step one includes: The benchmark visual model is forward-inferred on a test set formed by real data to identify all sample regions where the benchmark visual model fails to forward-infer within the test set range, and the failure mask is obtained. The physical feature vector of each failure mask is extracted using a calibrated digital twin simulation platform. The physical feature vector includes geometric features, lighting features, material features, and camera features. Cluster the physical feature vectors of all failed samples, and use the clustering to summarize the physical scene patterns of the failure of the benchmark visual model.

3. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, The targeted generation of physical perception training data based on the physical scene patterns obtained in Step 1 includes: obtaining the physical parameter ranges of each physical scene pattern summarized in Step 1; and using the PCG engine and a calibrated simulator to generate targeted physical perception training data rich in difficult samples. The physical perception training data adopts the following triplet format: simulation image. physical parameters The true value of the label corresponding to the simulated image .

4. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, The modulation parameter generation branch includes two multilayer perceptrons. The first multilayer perceptron maps physical parameters into higher-dimensional hidden physical parameter features. The second multilayer perceptron converts the hidden physical parameter features into adaptive modulation parameters, which include a scaling modulation vector and a bias modulation vector. A modulation module is disposed between the encoder and the decoder, and the dimensions of the two modulation vectors are consistent with the number of channels of the image feature map output by the encoder. The modulation module uses the modulation vectors to modulate the deepest image feature map of the encoder.

5. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, The modulation parameter generation branch includes a multilayer perceptron that maps physical parameters into higher-dimensional hidden physical parameter features. Several encoder and decoder units are used to convert the hidden physical parameter features into adaptive modulation parameter multilayer perceptrons. The adaptive modulation parameters include scaling modulation vectors and bias modulation vectors that adapt to the image feature maps of each encoder and decoder unit. The dimension of the modulation vector corresponding to each encoder and decoder unit level is consistent with the number of channels in the image feature map output by the encoder unit at that level.

6. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 5, characterized in that, The modulation parameter generation branch sets up several modulation modules corresponding to the encoder unit and decoder unit layers. The modulation modules are set between the encoder unit and decoder unit layers at the same level. Each modulation module modulates the image feature map output by each encoder unit using corresponding adaptive modulation parameters.

7. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 4 or 6, characterized in that, The modulation module receives the image feature map x from the encoder and the scaled modulation vector from the modulation parameter generation branch. and bias modulation vector ; Use scaling modulation vector and bias modulation vector Apply an affine transformation to each channel of the image feature map x, and the calculation formula is as follows: ； in: This represents element-wise multiplication; the modulation vector is scaled using an affine transformation. and bias modulation vector Adaptively adjust the activation values ​​of the image feature map.

8. The visual model closed-loop enhancement method based on physical simulation and parameter awareness according to claim 1, characterized in that, When training the improved baseline visual model end-to-end on the physical perception training data, the training objective is to minimize the difference between the baseline visual model output and the corresponding label ground truth.

9. A visual model closed-loop enhancement device based on physical simulation and parameter perception, comprising: At least one processing unit is connected to a storage unit via a bus unit, characterized in that the storage unit stores a computer program, and the processing unit implements the visual model closed-loop enhancement method based on physical simulation and parameter perception as described in any one of claims 1-8 by running the computer program stored in the storage unit.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the visual model closed-loop enhancement method based on physical simulation and parameter perception as described in any one of claims 1-8.

Images