Simulation-to-reality calibration method based on multi-modal loss and dynamic weighting

Abstract

The invention relates to a simulation-to-reality calibration method based on multi-modal loss and dynamic weighting. The method comprises the following steps: constructing a multi-modal and field-adaptive initial loss function; and generating a spatial weight map and a multi-scale spatial weight map for performing spatial dynamic weighting on the constructed multi-modal and field-adaptive initial loss function according to a position relationship between pixel regions of interest. Weighting a basic loss item of the initial loss function by the spatial weight map and the multi-scale spatial weight map; integrating the spatial dynamic weighted L1 loss, the spatial dynamic weighted multi-scale structure similarity loss, the perception loss and a target loss function after space-time weighting calculated by a weight scheduler; the weight scheduler is responsible for dynamically adjusting weights of different loss items according to training time in an optimization process; and calculating a gradient of simulation parameters of the simulator according to the target loss function, and iteratively updating the simulator parameters by using an optimizer according to the gradient so as to enable the target loss function to be minimum.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of graphic modeling, and in particular to a simulation-to-reality calibration method based on multi-modal loss and dynamic weighting. BACKGROUND

[0002] High-fidelity digital twinning is a basic technology in modern industry, scientific research and entertainment. In order to make the digital twinning system accurately reflect the physical entity, the simulation parameters must be calibrated so that the output simulation image matches the real image of the sensor data in the real world. This process is usually constructed as an optimization problem, that is, to find the optimal parameters by minimizing the "difference" between simulation and reality.

[0003] Currently, in the simulation-to-reality calibration of fields such as robots and autonomous driving, the most common technical solution is an optimization method based on a fixed hybrid loss function. The implementation scheme is usually as follows: a parameterized simulator is constructed, a static hybrid loss function composed of a weighted sum of multiple fixed loss functions with explicit mathematical formulas (such as L1 loss, structural similarity SSIM) is defined. A general optimizer (such as Adam) is selected to minimize the static hybrid loss function as the goal to optimize the parameters of the simulator. The L1, SSIM and other loss functions used in the existing scheme are essentially "low-level" metrics and are not sensitive to the "spirit" level differences in high-level textures, material textures and other aspects that can be perceived by humans or AI models. The simulator lacks perceptual ability; even if a general perceptual loss such as LPIPS is introduced, the feature extractor is trained on a general image library (such as ImageNet), and the ability to capture subtle features in a specific domain is limited, lacking domain specificity; the fixed loss weight cannot adapt to the dynamic changes from "macro alignment" to "detail carving" in the optimization process; all regions of the image are treated equally, resulting in scattered optimization "firepower", low efficiency, and rigid optimization strategy. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, in view of the above shortcomings, the purpose of the present application is to provide a simulation-to-reality calibration method based on multi-modal loss and dynamic weighting, which is more intelligent, more efficient and more perceptual. This method introduces a domain-adaptive perceptual loss and a spatiotemporal dynamic weighting strategy, allowing the optimization process to "focus on the key points", thereby achieving higher fidelity, especially in the perception level, in a shorter time.

[0005] In a first aspect, the present application provides a simulation-to-reality calibration method based on multi-modal loss and dynamic weighting, comprising: S100, constructing a multi-modal and domain adaptive initial loss function; the initial loss function of the application comprises a basic loss and a perception loss; the basic loss comprises an L1 loss and a multi-scale structural similarity loss; S200, generating a spatial weight map and a multi-scale spatial weight map for spatial dynamic weighting of the constructed multi-modal and domain adaptive initial loss function according to the positional relationship between the pixel regions of interest; the spatial weight map and the multi-scale spatial weight map respectively weight the L1 loss and the multi-scale structural similarity loss; S300, constructing a weight scheduler that dynamically changes with training time, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function during optimization according to training time; S400, integrating the spatially dynamically weighted L1 loss, the spatially dynamically weighted multi-scale structural similarity loss, the perception loss, and the spatio-temporally weighted target loss function calculated by the weight scheduler; S500, calculating the gradient of the simulation parameters of the simulator according to the target loss function, and updating the simulator parameters iteratively using an optimizer according to the gradient, so as to minimize the target loss function.

[0006] Further, the process of constructing the initial loss function is as follows: S110, calculating the L1 loss and the multi-scale structural similarity loss between the simulation image and the corresponding real image to obtain the basic loss; S120, constructing a self-supervised perception loss using a pre-trained variational autoencoder.

[0007] Further, constructing a self-supervised perception loss using a pre-trained variational autoencoder comprises: using unlabeled real images to pre-train a variational autoencoder, the variational autoencoder comprising an encoder and a decoder; After completing the pre-training, the encoder of the variational autoencoder is obtained, and the encoder weight of the variational autoencoder is frozen as a domain expert encoder; In the simulation-to-reality calibration process, the domain expert encoder acts as a fixed 'feature extractor' to convert the input simulation and real images into feature vectors in the latent space; The distance between the feature vectors in the latent space is taken as the perception loss between the simulation image and the real image.

[0008] Further, when training the variational autoencoder, a real image is input into the encoder of the variational autoencoder to generate a mean vector and a log variance vector in the form of a feature vector distribution representing the latent space; then a feature vector of the latent space is sampled from the mean vector and the log variance vector and input into the decoder of the variational autoencoder to reconstruct the original real image; the objective of training the variational autoencoder is to minimize the reconstruction loss and the KL divergence, the reconstruction loss being the difference between the original real image and the reconstructed image, and the KL divergence being the difference between the distribution generated by the encoder and the standard normal distribution.

[0009] Further, S200 comprises: S210, generating a binary region of interest mask through a segmentation model or prior knowledge The region of interest mask has the same size as the image; S220, defining a basic weight lifting coefficient The basic weight lifting coefficient The basic weight lifting coefficient represents how many times the importance of the region of interest is lifted; S230, generating a spatial weight map with the same size as the image using the region of interest mask and the basic weight lifting coefficient Each element of the spatial weight map is calculated as follows: ; is the value of the pixel position in the region of interest, when the pixel is in the region of interest roi, According to the formula, at this time ; when the pixel is outside the region of interest roi, According to the formula, at this time ; S240, the spatial weight map with the same size as the image is weighted by the L1 loss function to obtain a spatial dynamic weighted L1 loss; S250, performing multi-scale processing on the region of interest mask to obtain multi-scale region of interest masks corresponding to multi-scale images; S260, for any one scale region of interest mask, a spatial weight map corresponding to the scale is generated using the basic weight lifting coefficient: ; Wherein, is the pixel coordinate on the scale m image, and the spatial weight maps of all scales form a multi-scale spatial weight map; S270, calculating the spatial dynamic weighted multi-scale structural similarity loss by using the multi-scale spatial weight map.

[0010] Further, the spatial weight map The spatial dynamic weighted L1 loss is calculated as follows: ; wherein, The spatial dynamic weighted L1 loss makes the optimizer in the simulator training process focus on the error generated in the region of interest.

[0011] Further, calculating the spatial dynamic weighted multi-scale structural similarity loss by using the multi-scale spatial weight map comprises: The spatial weight map of scale m provides the weight for weighted average, and calculates the weighted average contrast and structure indicators: ; ; wherein, represents the image block centered on the pixel on the image of scale m; is the contrast comparison between the image blocks of the image of scale m; is the structure comparison between the image blocks of the image of scale m; m = 1, 2, …, M; M is the number of scales; Substitute the weighted average contrast and structure indicators into the multi-scale structural similarity formula between the multi-scale image blocks x and y containing the same content, to obtain the spatial weighted multi-scale structural similarity between the multi-scale image blocks x and y containing the same content: ; The spatial dynamic weighted multi-scale structural similarity between the real image and the simulation image : ; Then, the spatial dynamic weighted multi-scale structural similarity loss between the real image and the simulation image is: .

[0012] Further, the weight scheduler is constructed as follows, comprising: S310, setting the total optimization step number of the simulator as , the current is the t-th step, t is from 0 to ; S320, constructing a weight scheduler, the weight scheduler is designed for a basic loss and a perception loss respectively and ; wherein the basic loss time weight changes with training steps as follows: ; decreases linearly from an initial value 1.0 to 0 as the training steps increase; the perception loss time weight changes with training steps as follows: ; the perception loss time weight increases linearly from 0 to a final value 1.0 as the training steps increase; in the early stage of optimization, compared with is greater, the contribution of the basic loss to the initial loss function is greater than that of the perception loss, and the macro structure and brightness of the aligned image are aligned under the action of the basic loss; in the later stage of optimization, t gradually approaches T max , gradually approaches 0 as t increases, approaches 1.0, the contribution of the perception loss to the initial loss function is greater than that of the basic loss, and the perception details are modeled under the action of the perception loss.

[0013] In a second aspect, the present application provides a simulation-to-reality calibration device based on multi-modal loss and dynamic weighting, comprising: at least one processing unit, the processing unit is connected to a storage unit through a bus unit, the storage unit stores a computer program, the processing unit realizes the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting by running the computer program stored in the storage unit.

[0014] In a third aspect, the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed to realize the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting.

[0015] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: By introducing the self-supervised perception loss of domain adaptation, the present application can capture subtle textures and textures specific to the target scene that may be ignored by the general perception loss (LPIPS), so that the final calibration result is more realistic in the perception level, and ensures that the fidelity of the simulator in simulation-to-reality is higher.

[0016] The present application generates a spatial weight map and a multi-scale spatial weight map that dynamically weights the constructed multi-modal, domain adaptive initial loss function according to the positional relationship between the pixel regions of interest; the spatial weight map and the multi-scale spatial weight map respectively weight the L1 loss and the multi-scale structural similarity loss; a weight scheduler that dynamically changes with the training time is constructed, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function during the simulator optimization process according to the training time; the dynamic weighting of the present application concentrates computing resources in the most important image regions through spatial weighting of regions of interest; through time weight scheduling, it guides the optimization process to follow an efficient path of 'from coarse to fine'. The combination of the two can significantly accelerate the convergence speed and achieve higher fidelity in a shorter time, making the optimization efficiency of the model higher.

[0017] The present application is not a "one-size-fits-all" scale, but an intelligent calibration framework that "understands the key points and progresses gradually", which can better adapt to the optimization needs of complex scene simulators. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the description.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0020] Figure 1 A flowchart of a simulation-to-reality calibration method based on multi-modal loss and dynamic weighting provided by an embodiment of the present application; Figure 2 A flowchart of a perceptual loss constructed by a variational autoencoder provided by an embodiment of the present application; Figure 3 A flowchart of a spatial weight map and a multi-scale spatial weight map that are spatially dynamically weighted, and a flowchart of the spatial weight map and the multi-scale spatial weight map respectively weighting the L1 loss and the multi-scale structural similarity loss provided by an embodiment of the present application; Figure 4 A schematic diagram of a simulation-to-reality calibration device based on multi-modal loss and dynamic weighting provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0022] It should be noted that in this document, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the sentence "comprises a" does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0023] Embodiment 1 The present application aims to solve the technical problems of insufficient model robustness and poor generalization ability of existing deep learning visual models when dealing with real-world complex scenes, which are caused by the fact that training data cannot cover all physical "corner cases". In short, it is to solve the problem that AI models lack "physical common sense" and are "very fragile" in the face of complex reality.

[0024] In view of the above-mentioned shortcomings, the present application aims to provide a new, closed-loop, physically-aware simulation-to-reality calibration method based on multi-modal loss and dynamic weighting, comprising: S100, constructing a multi-modal, domain-adaptive initial loss function for training the simulator; the initial loss function of the present application comprises a basic loss and a perception loss. The basic loss comprises an L1 loss and a multi-scale structural similarity loss.

[0025] In the specific implementation process, the simulator generates a simulation image under the constraint of simulation parameters , assuming that the simulation image and the corresponding real image have an image height of H and a width of W. The generated simulation image is close to the real image.

[0026] The process of constructing the initial loss function is as follows: S110, calculating the L1 loss and the multi-scale structural similarity loss between the simulation image and the corresponding real image .​

[0027] wherein the L1 loss calculates the average of the sum of the absolute values of the pixel-wise difference between the simulated image and the real image The L1 loss is not sensitive to outliers and focuses on the overall brightness difference. The mathematical form of the L1 loss is: ; wherein are the pixel values of the real image and the simulated image at coordinate (i, j), respectively.

[0028] To compare the difference between the real image and the simulated image more sensitively and take into account the global and local details, the present application improves the structural similarity (SSIM) loss and constructs a multi-scale structural similarity loss.

[0029] The structural similarity compares the consistency of two image blocks and image block from three dimensions of brightness, contrast and structure, wherein image block and image block refer to the local image regions extracted from the same spatial position of the real image and the simulated image respectively. In actual calculation, a sliding window (for example, a sliding window with a size of 11x11) is used to traverse the real image and the simulated image . At each sliding window position, the center coordinate is (i, j), the real image content in the sliding window is image block , and the simulated image content is image block . By calculating the structural similarity scores of the image blocks at all positions and taking the average, the structural similarity of the entire image is finally obtained.

[0030] The brightness comparison between two image blocks and image block is as follows: ; The brightness comparison measures whether the average brightness of the two image blocks is close. When , reaches a maximum value of 1, indicating that the brightness of the two image blocks is completely consistent; the greater the difference between the two image blocks, the smaller the value.

[0031] The contrast comparison between two image blocks and image block is as follows: ; The contrast comparison measures whether the contrast of two image blocks is similar. When , reaches the maximum value 1, indicating that the contrast of two image blocks is completely consistent; the greater the difference between the contrast of two image blocks, the smaller the value.

[0032] The structure comparison between two image blocks and image block is as follows: ; wherein: are the pixel mean values in image block and image block respectively; are the pixel standard deviations in image block and image block respectively; are the pixel variances of image block and image block respectively; is the covariance of image block and image block , indicating the linear correlation of the structures of image block and image block ; is a constant for avoiding zero denominator, , , L is the maximum value of the pixel value, ; L is the dynamic range of the pixel value (e.g. 255 for an 8-bit image), and K1=0.01, K2=0.03 in the example; The structure similarity between image block and image block is expressed as: ; are both set to 1; The multi-scale structure similarity loss is to calculate the structure similarity at multiple scales of the simulation and real images, i.e. setting the scale index from 1 to M, for any scale m of the image, both the contrast comparison and the structure comparison are performed, and only at the last scale M, the brightness comparison is performed.

[0033] The multi-scale structure similarity between the multi-scale image block x and the image block y containing the same content is: ; wherein, All are set to 1; are two image blocks of scale m, m=1,2,...M.

[0034] Real images and simulation images The multi-scale structural similarities between them are as follows: ; Then, the real image and simulation images The multi-scale structural similarity loss between .

[0035] The multi-scale structural similarity loss is more consistent with human visual perception. When observing an image, the human eye focuses on both global structure (at low resolution) and local details (at high resolution). By calculating similarity at different scales, the multi-scale structural similarity loss better simulates how humans perceive image quality. Large-scale features tend to capture overall image structure (such as object shape), while small-scale features tend to capture image details such as texture and edges. The multi-scale structural similarity loss uses multi-scale structural comparison analysis to provide a more comprehensive assessment of image quality. At low resolutions, it relies on contrast and structure for comparison, prioritizing the overall image layout and matching local features. Luminance is calculated at the highest scale to avoid interference from low-resolution luminance information. It is applicable to different distortion types and is more sensitive to blur, noise, and compression artifacts, enabling differentiation between different types of distortion. For example, blur significantly degrades structural similarity at high resolutions, while noise affects both contrast and structure at multiple scales.

[0036] S120, uses the pre-trained variational autoencoder to construct a self-supervised perceptual loss. In the specific implementation process, Figure 2 As shown, including: A variational autoencoder (VAE) is pre-trained using a large number of unlabeled real-world images. The VAE consists of an encoder and a decoder. During VAE training, a real-world image is fed into the VAE's encoder, generating a mean vector and log-variance vector representing the distribution of feature vectors in the latent space. Feature vectors from the latent space are then sampled from the mean and log-variance vectors and fed into the VAE's decoder to reconstruct the original real-world image. The goal of VAE training is to minimize the reconstruction loss, which is the difference between the original real-world image and the reconstructed image, and the KL divergence, which is the difference between the distribution generated by the encoder and the standard normal distribution.

[0037] The encoder of the variational autoencoder is taken as a domain expert encoder; the training of the variational autoencoder is an independent, offline pre-training process, and the domain expert encoder learns a feature space that can deeply understand the data distribution of'real objects' in real images.

[0038] The simulation image is input into the domain expert encoder to obtain the corresponding feature vectors of the two images in the latent space . and , wherein, is the feature vector of the latent space of the simulation image: , is the feature vector of the latent space of the real image: . After the pre-training is completed, only the encoder of the variational autoencoder is taken, and the weights of the encoder are frozen to obtain the domain expert encoder.

[0039] In the simulation-to-reality calibration process, the domain expert encoder acts as a fixed 'feature extractor' to convert the input simulation and real images into feature vectors in the latent space.

[0040] The distance between the feature vectors in the latent space is taken as the perceptual loss between the simulation image and the real image: wherein is the perceptual similarity loss.

[0041] S110 and S120 do not represent the order of technical features, but are only used to divide the content.

[0042] S200, according to the positional relationship between the pixel regions of interest, a spatial weight map and a multi-scale spatial weight map are generated for spatially dynamically weighting the initial loss function constructed in a multi-modal and domain adaptive manner. The spatial weight map and the multi-scale spatial weight map weight the basic loss term of the initial loss function.

[0043] As shown in Figure 3 , the specific method is as follows: S210, a binary region-of-interest mask is generated by a segmentation model or priori knowledge. The size of the region-of-interest mask is the same as that of the image.

[0044] S220, a basic weight lifting coefficient is defined. For example, The base weight boost factor represents how many times the importance of the region of interest is boosted, and is set to 2, 5 or 10.

[0045] S230, generating a spatial weight map of the same size as the image using the region of interest mask and the base weight boost factor The spatial weight map Each element of the spatial weight map ; is calculated as follows: When the pixel is within the region of interest roi, According to the formula, at this time When the pixel is outside the region of interest roi, According to the formula, at this time

[0046] S240, the L1 loss function needs to be calculated pixel by pixel, and the spatial weight map of the same size as the image The spatial dynamic weighted L1 loss is calculated as follows: ; The spatial dynamic weighted L1 loss makes the optimizer in the simulator training process pay more attention to the error generated in the region of interest.

[0047] S250, in the weighted multi-scale structural similarity, the region of interest mask is multi-scale processed to obtain a multi-scale region of interest mask corresponding to the multi-scale image, and the multi-scale region of interest mask is used to give a higher weight to the region of interest of each scale compared to the non-region of interest.

[0048] In the implementation process, the region of interest mask is down-sampled step by step to generate a multi-scale mask pyramid. For example, the average pooling or Gaussian pyramid down-sampling with a step of 2 obtains a multi-scale mask pyramid, the multi-scale mask pyramid corresponds to a multi-scale image, contains a plurality of multi-scale region of interest masks, and the mth scale region of interest mask is: , wherein is down-sampling.

[0049] S260, for any one scale region of interest mask , a spatial weight map of the corresponding scale is generated using the base weight boost factor: ; , wherein is the pixel coordinate on the scale m image. The spatial weight map of all scales A multi-scale spatial weight map is formed.

[0050] S270, the spatial dynamic weighted multi-scale structural similarity loss is calculated using the multi-scale spatial weight map. As mentioned before, the calculation of multi-scale structural similarity involves the evaluation of contrast and structure at multiple scales respectively. When calculating the structural similarity at an arbitrary scale m, the spatial weight map of scale m The weights of the weighted average are provided, and the weighted average contrast and structure indicators are calculated: ; ; wherein, represents the image block centered at the pixel on the scale m image; is the contrast comparison between the image blocks of the scale m image; is the structure comparison between the image blocks of the scale m image.

[0051] The weighted average contrast and structure indicators are substituted into the multi-scale structural similarity formula between the multi-scale image blocks x and y containing the same content, to obtain the spatial weighted multi-scale structural similarity between the multi-scale image blocks x and y containing the same content: ; The spatial dynamic weighted multi-scale structural similarity between the real image and the simulation image : ; Then, the spatial dynamic weighted multi-scale structural similarity loss between the real image and the simulation image is: .

[0052] The spatial dynamic weighted multi-scale structural similarity loss enables the optimizer to pay more attention to the image blocks within the region of interest at each scale when calculating the structural similarity, thereby achieving precise spatial dynamic weighting.

[0053] S300, a weight scheduler that dynamically changes with the training time is constructed, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function according to the training time during the optimization process, so as to realize the 'from coarse to fine' optimization strategy by using dynamic time scheduling.

[0054] The specific implementation is as follows: The total optimization step number of the simulator is set to , the current step is t, t ranges from 0 to .

[0055] Construct a weight scheduler that designs time weights for basic loss and perceptual loss respectively and .

[0056] Among them, the basic loss time weight The changes with training steps are as follows: ; As the training step increases, it decreases linearly from an initial value of 1.0 to 0.

[0057] Perceived loss time weight The changes with training steps are as follows: ; Perceived loss time weight As the number of training steps increases, it increases linearly from 0 to a final value of 1.0.

[0058] In the early stage of optimization, t is smaller than that in the later stage of optimization. Comparison Bigger, Comparison Smaller, the contribution of the base loss to the initial loss function is greater than the perceptual loss. Under the action of the base loss, the system focuses on aligning the macro structure and brightness of the image. In the later stage of optimization, t is close to T max , tends to 0, It tends to 1.0, and the contribution of perceptual loss to the initial loss function is greater than that of basic loss. Under the influence of perceptual loss, the system turns to fine-tuning high-level perceptual details such as materials, light and shadow.

[0059] S400 is the target loss function after spatial and temporal weighting calculated by integrating spatial dynamic weighted L1 loss, spatial dynamic weighted multi-scale structural similarity loss, perceptual loss and weight scheduler. The target loss function is expressed as follows: ; is the target loss function, which is the final scalar value used by the optimizer to calculate the gradient and update the simulation parameter θ; It is the spatial dynamic weighted L1 loss; A spatially dynamically weighted multi-scale structural similarity loss.

[0060] S500, calculate the gradient of the simulation parameter of the simulator according to the target loss function, and update the simulation parameter of the simulator according to the gradient iteration using the optimizer, so as to minimize the target loss function.

[0061] There are alternatives for the specific technical modules of the present application, which should also fall within the protection scope of the present application: On the optimizer: in addition to the gradient-based optimizer (such as Adam); for more complex black-box simulation models, gradient-independent optimization algorithms such as Bayesian optimization, genetic algorithm or particle swarm optimization algorithm can also be used as alternatives.

[0062] In the measurement of perceptual difference: in addition to using the variational autoencoder, the framework of the generative adversarial network is used to obtain the feature vector representing the perception through the encoder of the generative adversarial network, and the statistical distribution difference between the simulation image and the real image is measured and narrowed down in the manner of adversarial training.

[0063] In the simulation modeling: for complex components such as lighting or cameras, in addition to using the analytical model based on physical equations, a data-driven learnable scene representation model can also be used as an alternative, as long as the data-driven learnable scene representation model is parameterized and can be optimized. The data-driven learnable scene representation model includes but is not limited to: neural radiance field (NeRF) and its variants based on continuous function implicit representation, which use the weights of the neural network to implicitly represent the scene's radiance and density field. 3D Gaussian splatting model based on discrete primitive explicit representation, which uses a large number of parameterized Gaussian bodies to explicitly construct the scene.

[0064] Embodiment 2 Referring to Figure 4 As shown in the drawings, the embodiment of the present application provides a simulation-to-reality calibration system based on multi-modal loss and dynamic weighting, which comprises at least one processing unit connected to a storage unit through a bus unit, and the storage unit serves as a computer readable storage medium and can be used to store software programs, computer executable programs and modules, such as the software programs, computer executable programs and modules corresponding to the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting in the embodiment of the present application. The processing unit realizes the above-mentioned simulation-to-reality calibration method based on multi-modal loss and dynamic weighting by running the software programs, computer executable programs and modules stored in the storage unit, which comprises: S100, construct a multi-modal, domain-adaptive initial loss function; the initial loss function of the present application includes a basic loss and a perception loss; the basic loss includes an L1 loss and a multi-scale structural similarity loss; S200, generate a spatial weight map for spatially dynamically weighting the constructed multi-modal, domain-adaptive initial loss function according to the positional relationship between the pixel regions of interest; S300, construct a weight scheduler that dynamically changes with training time, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function according to training time during the optimization process; S400, calculate a spatio-temporal weighted target loss function by comprehensively considering the spatial weight map, the weight scheduler, and the initial loss function; S500, calculate the gradient of the simulation parameters of the simulator according to the target loss function, and update the simulator parameters iteratively using an optimizer according to the gradient, so as to minimize the target loss function.

[0065] Of course, the storage unit in the simulation-to-reality calibration system based on multi-modal loss and dynamic weighting provided by the embodiments of the present application stores a computer program, which is not limited to the method operations described above, but can also perform related operations in the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting provided by any embodiment of the present application.

[0066] Embodiment 3 The embodiments of the present application provide a computer readable storage medium storing a computer program, which, when executed, implements the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting, comprising: S100, construct a multi-modal, domain-adaptive initial loss function; the initial loss function of the present application includes a basic loss and a perception loss; the basic loss includes an L1 loss and a multi-scale structural similarity loss; S200, generate a spatial weight map for spatially dynamically weighting the constructed multi-modal, domain-adaptive initial loss function according to the positional relationship between the pixel regions of interest; S300, construct a weight scheduler that dynamically changes with training time, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function according to training time during the optimization process; S400, calculate a spatio-temporal weighted target loss function by comprehensively considering the spatial weight map, the weight scheduler, and the initial loss function; S500, calculate the gradient of the simulation parameters of the simulator according to the target loss function, and update the simulator parameters iteratively using an optimizer according to the gradient, so as to minimize the target loss function.

[0067] The computer program stored in the computer readable storage medium provided by the embodiment of the present application is not limited to the method operations described above, and can also perform related operations in the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting provided by any embodiment of the present application.

[0068] In the embodiments provided by the present application, it should be understood that the disclosed structure and method can be implemented in other ways. For example, the above-described structural embodiments are merely illustrative. For example, the division of the units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interface, structure or unit, and can be electrical, mechanical or in other forms.

[0069] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0070] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0071] The above is only a specific implementation of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A simulation-to-reality calibration method based on multimodal loss and dynamic weighting, characterized in that: include: S100, constructing a multimodal, domain-adaptive initial loss function for training the simulator; the initial loss function includes a basic loss and a perceptual loss; the basic loss includes an L1 loss and a multi-scale structural similarity loss; S200, generating a spatial weight map and a multi-scale spatial weight map for spatially dynamically weighting the constructed multimodal, domain-adaptive initial loss function according to the positional relationship between pixel regions of interest; the spatial weight map and the multi-scale spatial weight map are weighted by L1 loss and multi-scale structural similarity loss respectively; S300, constructing a weight scheduler that changes dynamically with training time, wherein the weight scheduler is responsible for dynamically adjusting the weights of different loss terms in the initial loss function according to the training time during the simulator optimization process; S400, the spatially weighted objective loss function calculated by integrating spatial dynamic weighted L1 loss, spatial dynamic weighted multi-scale structural similarity loss, perceptual loss and weight scheduler; S500, calculating the gradient of the simulation parameters of the simulator according to the target loss function, and using the optimizer to iteratively update the simulator parameters according to the gradient to minimize the target loss function.

2. The simulation-to-reality calibration method based on multimodal loss and dynamic weighting according to claim 1, characterized in that: The process of constructing the initial loss function is as follows: S110, calculation simulation image and the corresponding real image The L1 loss and multi-scale structural similarity loss between them are used to obtain the basic loss; S120, constructs self-supervised perceptual loss using pre-trained variational autoencoders.

3. The simulation-to-reality calibration method based on multimodal loss and dynamic weighting according to claim 2, characterized in that: The self-supervised perceptual loss constructed using a pre-trained variational autoencoder includes: Use unlabeled real images to pre-train a variational autoencoder, which consists of an encoder and a decoder; After pre-training, obtain the encoder of the variational autoencoder and freeze the encoder weights of the variational autoencoder as a domain expert encoder; In the simulation-to-reality calibration process, the domain expert encoder acts as a fixed 'feature extractor' that converts the input simulated and real images into feature vectors in the latent space; The distance between the feature vectors in the latent space is used as the perceptual loss between the simulated image and the real image.

4. The simulation-to-reality calibration method based on multimodal loss and dynamic weighting according to claim 3, characterized in that: When training a variational autoencoder, a real image is input into the variational autoencoder's encoder, generating a mean vector and a logarithmic variance vector that represent the distribution of the feature vectors in the latent space. The feature vectors of the latent space are then sampled from the mean vector and the logarithmic variance vector and fed into the variational autoencoder's decoder to reconstruct the original real image. The goal of training a variational autoencoder is to minimize the reconstruction loss and the KL divergence, where the reconstruction loss is the difference between the original real image and the reconstructed image, and the KL divergence is the difference between the distribution produced by the encoder and the standard normal distribution.

5. The simulation-to-reality calibration method based on multimodal loss and dynamic weighting according to claim 1, characterized in that: S200 includes: S210, generate a binary region of interest mask through segmentation model or prior knowledge , region of interest mask The same size as the image; S220, define a basic weight improvement coefficient , basic weight improvement coefficient ,The basic weight improvement coefficient represents how many times the importance of the region of interest is improved; S230, using the region of interest mask and the basic weight boost coefficient to generate a spatial weight map as large as the image , the spatial weight map Each element of is calculated as follows: ； is the value of the pixel position in the region of interest. When in the region of interest (ROI), , according to the formula, at this time ; When the pixel When outside the region of interest roi, , according to the formula, at this time ; S240, a spatial weight map with the same size as the image The weighted L1 loss function obtains the spatial dynamic weighted L1 loss; S250, performing multi-scale processing on the region of interest mask to obtain a multi-scale region of interest mask corresponding to the multi-scale image; S260, for any scale of interest region mask, the basic weight lifting coefficient is used to generate a spatial weight map of the corresponding scale : ； in, is the pixel coordinate on the scale m image, the spatial weight map of all scales Forming a multi-scale spatial weight map; S270, using the multi-scale spatial weight map to calculate the spatial dynamic weighted multi-scale structural similarity loss.

6. The simulation-to-reality calibration method based on multimodal loss and dynamic weighting according to claim 5, characterized in that: The spatial weight map The weighted L1 loss function is used to obtain the spatial dynamic weighted L1 loss. The calculation is as follows: ； in, The spatially dynamically weighted L1 loss allows the optimizer of the simulator training process to focus on the errors generated in the region of interest.

7. The simulation-to-reality calibration method based on multimodal loss and dynamic weighting according to claim 5, characterized in that: Calculating the spatial dynamic weighted multi-scale structural similarity loss using a multi-scale spatial weight map includes: Spatial weight map of scale m Provide weights for weighted averaging and calculate weighted average contrast and structure metrics: ； ； in, Represents the pixels on the image at scale m The image patch centered on is the image patch of scale m image Contrast comparison between is the image patch of scale m image Structural comparison between them; m=1,2,...M; M is the number of scales; Substituting the weighted average contrast and structural index into the multi-scale structural similarity formula between multi-scale image patches x and image patches y with the same content, we can obtain the spatial weighted multi-scale structural similarity between multi-scale image patches x and image patches y with the same content: ； Real images and simulation images Spatial dynamic weighted multi-scale structural similarity between: ； Then, the real image and simulation images The spatial dynamic weighted multi-scale structural similarity loss between is: .

8. The simulation-to-reality calibration method based on multimodal loss and dynamic weighting according to claim 1, wherein: The weight scheduler is constructed as follows, including: S310, set the total number of optimization steps of the simulator to , the current step is t, t ranges from 0 to ; S320: Build a weight scheduler that designs time weights for basic loss and perceptual loss respectively. and ; Among them, the basic loss time weight The changes with training steps are as follows: ； As the training step increases, it decreases linearly from an initial value of 1.0 to 0; Perceived loss time weight The changes with training steps are as follows: ； Perceived loss time weight As the number of training steps increases, it increases linearly from 0 to a final value of 1.0; In the early stages of optimization, Comparison The contribution of the basic loss to the initial loss function is greater than that of the perceptual loss. Under the action of the basic loss, the macro structure and brightness of the image are aligned. In the later stage of optimization, t gradually approaches T max , As t increases, it gradually approaches 0. Tends to 1.0, the contribution of perceptual loss to the initial loss function is greater than that of basic loss, and under the action of perceptual loss, perceptual details are modeled.

9. A simulation-to-reality calibration device based on multimodal loss and dynamic weighting, comprising: At least one processing unit, wherein the processing unit is connected to a storage unit via a bus unit, wherein the storage unit stores a computer program, and the processing unit implements the simulation-to-reality calibration method based on multimodal loss and dynamic weighting as described in any one of claims 1-8 by running the computer program stored in the storage unit.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the simulation-to-reality calibration method based on multimodal loss and dynamic weighting described in any one of claims 1 to 8 is implemented.

Images