A simulation-to-reality calibration method based on multi-modal loss and dynamic weighting

Abstract

The present application relates to a kind of simulation to reality calibration method based on multi-modal loss and dynamic weighting, the present application constructs multi-modal, field adaptive initial loss function;According to the position relationship between pixel region of interest, generate the spatial weight map and multi-scale spatial weight map that space dynamically weight is carried out to the multi-modal, field adaptive initial loss function constructed;The basic loss term of initial loss function is weighted by spatial weight map and multi-scale spatial weight map;Integrate spatial dynamic weighting L1 loss, spatial dynamic weighting multi-scale structural similarity loss, perception loss and the target loss function after spatiotemporal weighting calculated by weight scheduler;Weight scheduler is responsible for in optimization process, according to training time, dynamically adjust the weight of different loss term;According to the gradient of the simulation parameter of simulator calculated by target loss function, and using optimizer to update simulator parameter according to gradient iteration, so that the target loss function is minimum.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of graphic modeling, and in particular to a simulation-to-reality calibration method based on multi-modal loss and dynamic weighting. BACKGROUND

[0002] High-fidelity digital twinning is a basic technology in modern industry, scientific research and entertainment. In order to make the digital twinning system accurately reflect the physical entity, the simulation parameters must be calibrated so that the output simulation image matches the real image of the sensor data in the real world. This process is usually constructed as an optimization problem, that is, to find the optimal parameters by minimizing the "difference" between simulation and reality.

[0003] Currently, in the simulation-to-reality calibration of fields such as robots and autonomous driving, the most common technical solution is an optimization method based on a fixed hybrid loss function. The implementation scheme is usually as follows: a parameterized simulator is constructed, a static hybrid loss function composed of a weighted sum of multiple fixed loss functions with explicit mathematical formulas (such as L1 loss, structural similarity SSIM) is defined. A general optimizer (such as Adam) is selected to minimize the static hybrid loss function as the goal to optimize the parameters of the simulator. The L1, SSIM and other loss functions used in the existing scheme are essentially "low-level" metrics and are not sensitive to the "spirit" level differences in high-level textures, material textures and other aspects that can be perceived by humans or AI models. The simulator lacks perceptual ability; even if a general perceptual loss such as LPIPS is introduced, the feature extractor is trained on a general image library (such as ImageNet), and the ability to capture subtle features in a specific domain is limited, lacking domain specificity; the fixed loss weight cannot adapt to the dynamic changes from "macro alignment" to "detail carving" in the optimization process; all regions of the image are treated equally, resulting in scattered optimization "firepower", low efficiency, and rigid optimization strategy. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, in view of the above shortcomings, the purpose of the present application is to provide a simulation-to-reality calibration method based on multi-modal loss and dynamic weighting, which is more intelligent, more efficient and more perceptual. This method introduces a domain-adaptive perceptual loss and a spatiotemporal dynamic weighting strategy, allowing the optimization process to "focus on the key points", thereby achieving higher fidelity, especially in the perception level, in a shorter time.

[0005] In a first aspect, the present application provides a simulation-to-reality calibration method based on multi-modal loss and dynamic weighting, comprising:

[0006] S100, constructing a multi-modal, domain-adaptive initial loss function; the initial loss function of the application comprises a basic loss and a perception loss; the basic loss comprises an L1 loss and a multi-scale structural similarity loss;

[0007] S200, generating a spatial weight map and a multi-scale spatial weight map for spatial dynamic weighting of the constructed multi-modal, domain-adaptive initial loss function according to the positional relationship between the pixel regions of interest; the spatial weight map and the multi-scale spatial weight map respectively weight the L1 loss and the multi-scale structural similarity loss;

[0008] S300, constructing a weight scheduler that dynamically changes with training time, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function during optimization according to training time;

[0009] S400, integrating the spatially dynamically weighted L1 loss, the spatially dynamically weighted multi-scale structural similarity loss, the perception loss, and the spatio-temporally weighted target loss function calculated by the weight scheduler;

[0010] S500, calculating the gradient of the simulation parameters of the simulator according to the target loss function, and updating the simulator parameters iteratively using an optimizer according to the gradient to minimize the target loss function.

[0011] Further, the process of constructing the initial loss function is as follows:

[0012] S110, calculating the L1 loss and the multi-scale structural similarity loss between the simulation image and the corresponding real image to obtain the basic loss;

[0013] S120, constructing a self-supervised perception loss using a pre-trained variational autoencoder.

[0014] Further, constructing a self-supervised perception loss using a pre-trained variational autoencoder comprises:

[0015] Using unlabeled real images to pre-train a variational autoencoder, the variational autoencoder comprising an encoder and a decoder;

[0016] After completing the pre-training, the encoder of the variational autoencoder is obtained, and the encoder weight of the variational autoencoder is frozen as a domain expert encoder;

[0017] In the simulation-to-reality calibration process, the domain expert encoder acts as a fixed 'feature extractor' to convert the input simulation and real images into feature vectors in the latent space;

[0018] The distance between the feature vectors in the latent space is taken as the perception loss between the simulation image and the real image.

[0019] Further, when training the variational autoencoder, the real image is input into the encoder of the variational autoencoder to generate a mean vector and a log variance vector in the form of a feature vector distribution representing the latent space; then the feature vector of the latent space is sampled from the mean vector and the log variance vector and input into the decoder of the variational autoencoder to reconstruct the original real image; the objective of training the variational autoencoder is to minimize the reconstruction loss and the KL divergence, the reconstruction loss being the difference between the original real image and the reconstructed image, and the KL divergence being the difference between the distribution generated by the encoder and the standard normal distribution.

[0020] Further, S200 comprises:

[0021] S210, generating a binary region of interest mask through a segmentation model or prior knowledge The region of interest mask has the same size as the image;

[0022] S220, defining a basic weight lifting coefficient The basic weight lifting coefficient The basic weight lifting coefficient represents how many times the importance of the region of interest is lifted;

[0023] S230, generating a spatial weight map with the same size as the image using the region of interest mask and the basic weight lifting coefficient Each element of the spatial weight map is calculated as follows:

[0024] ;

[0025] is the value of the pixel position in the region of interest, when the pixel is in the region of interest roi, According to the formula, at this time ; when the pixel is outside the region of interest roi, According to the formula, at this time ;

[0026] S240, the spatial weight map with the same size as the image is weighted by the L1 loss function to obtain a spatial dynamic weighted L1 loss;

[0027] S250, performing multi-scale processing on the region of interest mask to obtain a multi-scale region of interest mask corresponding to the multi-scale image;

[0028] S260, for any one scale region of interest mask, a spatial weight map corresponding to the scale is generated using the basic weight lifting coefficient :

[0029] ;

[0030] wherein, is the pixel coordinate on the scale m image, the spatial weight map of all scales to form the multi-scale spatial weight map;

[0031] S270, calculating the spatial dynamic weighted multi-scale structural similarity loss by using the multi-scale spatial weight map.

[0032] Further, the spatial weight map The spatial dynamic weighted L1 loss is calculated as follows:

[0033] ;

[0034] wherein, The spatial dynamic weighted L1 loss makes the error generated in the region of interest to be focused on by the optimizer in the simulator training process.

[0035] Further, calculating the spatial dynamic weighted multi-scale structural similarity loss by using the multi-scale spatial weight map includes:

[0036] The spatial weight map of scale m provides the weight of the weighted average, and calculates the weighted average contrast and structure indicators:

[0037] ;

[0038] ;

[0039] wherein, represents the image block centered on the pixel on the scale m image; is the contrast comparison between the image blocks of the scale m image; is the structure comparison between the image blocks of the scale m image; m=1,2,...M; M is the number of scales;

[0040] Substitute the weighted average contrast and structure indicators into the multi-scale structural similarity formula between the multi-scale image blocks x and y containing the same content, to obtain the spatial weighted multi-scale structural similarity between the multi-scale image blocks x and y containing the same content:

[0041] ;

[0042] The real image and the simulation image the spatial dynamic weighted multi-scale structural similarity between real image

[0043] ;

[0044] the spatial dynamic weighted multi-scale structural similarity between real image and simulation image is: .

[0045] Further, the weight scheduler is constructed as follows, including:

[0046] S310, setting the total optimization step number of the simulator as , the current is the t-th step, t from 0 to ;

[0047] S320, constructing a weight scheduler, the weight scheduler designs time weights and for the basic loss and the perception loss respectively;

[0048] wherein, the basic loss time weight changes with the training step as follows:

[0049] ;

[0050] with the increase of the training step, linearly decreases from an initial value 1.0 to 0;

[0051] the perception loss time weight changes with the training step as follows:

[0052] ;

[0053] the perception loss time weight increases from 0 to an ultimate value 1.0 linearly with the increase of the training step;

[0054] in the early optimization stage, is larger than , the contribution of the basic loss to the initial loss function is larger than that of the perception loss, and the macro structure and brightness of the aligned image are aligned under the action of the basic loss; in the later optimization stage, t gradually approaches T max , with the increase of t, gradually tends to 0, tends to 1.0, the contribution of the perception loss to the initial loss function is larger than that of the basic loss, and the perception details are modeled under the action of the perception loss.

[0055] In a second aspect, the present application provides a simulation-to-reality calibration device based on multi-modal loss and dynamic weighting, comprising: at least one processing unit, the processing unit being connected to a storage unit through a bus unit, the storage unit storing a computer program, and the processing unit realizing the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting by running the computer program stored in the storage unit.

[0056] In a third aspect, the present application provides a computer readable storage medium storing a computer program, the computer program being executed to realize the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting.

[0057] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art:

[0058] By introducing the self-supervised perception loss of domain adaptation, the present application can capture subtle textures and qualities specific to the target scene that may be ignored by the general perception loss (LPIPS), making the final calibration result more realistic in the perception level and ensuring that the simulator achieves higher fidelity in simulation-to-reality.

[0059] The present application generates a spatial weight map and a multi-scale spatial weight map for spatial dynamic weighting of the constructed multi-modal, domain-adaptive initial loss function according to the positional relationship between the pixel regions of interest; the spatial weight map and the multi-scale spatial weight map respectively weight the L1 loss and the multi-scale structural similarity loss; a weight scheduler that dynamically changes with the training time is constructed, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function during the optimization process of the simulator according to the training time; the dynamic weighting of the present application concentrates computing resources in the most important image regions through spatial weighting of regions of interest; through time weight scheduling, it guides the optimization process to follow an efficient path of "from coarse to fine". The combination of the two can significantly accelerate the convergence speed and achieve higher fidelity in a shorter time, making the optimization efficiency of the model higher.

[0060] The present application is not a "one-size-fits-all" ruler, but an intelligent calibration framework that "knows how to seize the key points and gradually progress", which can better adapt to the optimization needs of complex scene simulators. BRIEF DESCRIPTION OF DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0063] Figure 1 The flow chart of the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting provided by the embodiments of the present application;

[0064] Figure 2 The flow chart of the perceptual loss constructed by the variational autoencoder provided by the embodiments of the present application;

[0065] Figure 3 The spatial weight map and the multi-scale spatial weight map for spatial dynamic weighting provided by the embodiments of the present application; and the flow chart of the spatial weight map and the multi-scale spatial weight map respectively weighting the L1 loss and the multi-scale structural similarity loss;

[0066] Figure 4 The schematic diagram of the simulation-to-reality calibration device based on multi-modal loss and dynamic weighting provided by the embodiments of the present application. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the protection scope of the present application.

[0068] It should be noted that, in this document, the term “comprising” or “including” or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes the elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement “including a…” does not exclude the presence of another identical element in the process, method, article or device including the element.

[0069] Embodiment 1

[0070] The present application aims to solve the technical problems of insufficient model robustness and poor generalization ability of existing deep learning visual models when dealing with real-world complex scenes, which leads to a sharp decline in performance when facing low contrast, complex lighting, dynamic blur and other situations. In short, it is to solve the problem that AI models lack "physical common sense" and are "very fragile" in the face of complex reality.

[0071] In view of the above-mentioned defects, the purpose of the present application is to provide a brand-new, closed-loop, physically-aware simulation-to-reality calibration method based on multi-modal loss and dynamic weighting, comprising:

[0072] S100, constructing a multi-modal, domain-adaptive initial loss function for training the simulator; the initial loss function of the present application comprises a basic loss and a perception loss. The basic loss comprises an L1 loss and a multi-scale structural similarity loss.

[0073] In the specific implementation process, the simulator generates a simulation image under the constraint of simulation parameters . Assuming that the simulation image and the corresponding real image have an image height of H and a width of W. The generated simulation image is close to the real image.

[0074] The process of constructing the initial loss function is as follows:

[0075] S110, calculating the L1 loss and the multi-scale structural similarity loss between the simulation image and the corresponding real image .

[0076] Wherein, the L1 loss calculates the average value of the sum of absolute values of pixel-by-pixel differences between the simulation image and the real image . The L1 loss is not sensitive to outliers and focuses on overall brightness differences. The mathematical form of the L1 loss is:

[0077] ;

[0078] Wherein, are the pixel values of the real image and the simulation image at coordinates (i,j), respectively.

[0079] In order to be more sensitive and take into account global and local details, the present application improves the structural similarity (SSIM) loss to construct a multi-scale structural similarity loss.

[0080] Structural similarity compares two image blocks from three dimensions of brightness, contrast and structure and image blocks Consistency, image patches and image blocks They refer to the real images respectively and simulated images The local image regions extracted at the same spatial location are, in actual computation, traversed by a sliding window (e.g., an 11x11 sliding window) on the real image. and simulated images At each sliding window position, with center coordinates (i, j), the actual image content within the sliding window is the image patch. The content of the simulated image is the image block. By calculating the structural similarity scores of image patches at all locations and taking their average, the structural similarity of the entire image is finally obtained.

[0081] Two image blocks and image blocks The brightness comparison between them is as follows:

[0082] ;

[0083] Brightness comparison measures whether the average brightness of two image patches is similar. hour, Reaching a maximum value of 1 indicates that the brightness of the two image blocks is completely identical; the greater the difference between the two image blocks, the lower the brightness. The smaller the value.

[0084] Two image blocks and image blocks The contrast between them is compared as follows:

[0085] ;

[0086] Contrast comparison measures whether the contrast of two image patches is similar. When hour, A value of 1 indicates that the contrast of the two image patches is completely identical; the greater the difference in contrast between the two image patches, the lower the value. The smaller the value.

[0087] Two image blocks and image blocks The structures are compared as follows:

[0088] ;

[0089] in: Image blocks and image blocks The average pixel value in the image; Image blocks and image block ; are pixel variance of image block and image block , respectively; is the covariance of image block and image block , indicating the linear correlation of the structure of image block and image block ; is a constant to avoid zero denominator, , L is the maximum value of pixel value, ; L is the dynamic range of pixel value (e.g. 255 for 8-bit image), and K1=0.01, K2=0.03 in the example;

[0090] The structural similarity between image block and image block is expressed as:

[0091] ; are all set to 1;

[0092] The multi-scale structural similarity loss is to calculate the structural similarity at multiple scales of the simulation and real images, i.e. set the scale index from 1 to M, for any scale m of the image, both the contrast comparison and the structure comparison are performed, and only at the last scale M, the brightness comparison is performed.

[0093] The multi-scale structural similarity between the multi-scale image blocks x and y containing the same content is:

[0094] ;

[0095] wherein, are all set to 1; are two image blocks at scale m, m=1, 2,...M.

[0096] The multi-scale structural similarity between the real image and the simulation image is as follows: ;

[0097] Then, the multi-scale structural similarity loss between the real image and the simulation image is .

[0098] The multi-scale structural similarity loss is more consistent with human visual perception. When observing an image, the human eye will focus on both the global structure (low resolution) and the local details (high resolution). The multi-scale structural similarity loss calculates the similarity at different scales, which can better simulate the way humans perceive image quality. Large-scale features often capture the overall structure of an image (such as object shape), while small-scale features often capture image details such as texture and edges. The multi-scale structural similarity loss can more comprehensively evaluate image quality through multi-scale structural comparison. At a low resolution scale, the comparison relies on contrast and structure, focusing more on the overall layout and local feature matching of the image. At the highest scale, the brightness is calculated to avoid the interference of brightness information at low resolution. It is suitable for different distortion types and more sensitive to blurring, noise, and compression artifact distortion, supporting the differentiation of different types of distortion. For example, when blurring occurs, the structural similarity at high resolution scales will decrease significantly. When noise occurs, the contrast and structure will be affected at multiple scales.

[0099] In S120, a pre-trained variational autoencoder is used to construct a self-supervised perception loss. In the specific implementation process, as shown in Figure 2 , it includes:

[0100] A large amount of unlabeled real images are used to pre-train a variational autoencoder, which consists of an encoder and a decoder. During training of the variational autoencoder, real images are input into the encoder of the variational autoencoder to generate a mean vector and a log variance vector in the form of a feature vector distribution representing the latent space. Then, a feature vector of the latent space is sampled from the mean vector and the log variance vector and input into the decoder of the variational autoencoder to reconstruct the original real image. The goal of training the variational autoencoder is to minimize the reconstruction loss and the KL divergence. The reconstruction loss is the difference between the original real image and the reconstructed image, and the KL divergence is the difference between the distribution generated by the encoder and the standard normal distribution.

[0101] The encoder of the variational autoencoder is obtained as a domain expert encoder. The training of the variational autoencoder is an independent and offline pre-training process, and the domain expert encoder learns a feature space that can deeply understand the data distribution of'real objects' in real images.

[0102] The simulation image and the real image are input into the domain expert encoder to obtain the corresponding feature vectors of the two images in the latent space and ,

[0103] wherein, is the feature vector of the simulation image in the latent space: , Feature vector of latent space for real image: ;

[0104] After pre-training, only the encoder of the variational autoencoder is taken , and the weights of the encoder are frozen to obtain a domain expert encoder.

[0105] In the simulation-to-reality calibration process, the domain expert encoder acts as a fixed 'feature extractor' to convert the input simulation and real images into feature vectors of the latent space.

[0106] The distance between the feature vectors of the latent space is taken as the perceptual loss between the simulation image and the real image: , where is the perceptual similarity loss.

[0107] S110 and S120 do not represent the order of technical features, but only serve to divide the content.

[0108] S200, according to the positional relationship between the pixel regions of interest, generate a spatial weight map and a multi-scale spatial weight map for spatial dynamic weighting of the constructed multi-modal, domain adaptive initial loss function. The spatial weight map and the multi-scale spatial weight map weight the basic loss term of the initial loss function.

[0109] As shown in Figure 3 , the specific method is as follows:

[0110] S210, generate a binary region of interest mask by a segmentation model or prior knowledge The size of the region of interest mask is the same as the image.

[0111] S220, define a basic weight lifting coefficient , the basic weight lifting coefficient , for example, is set to 2, 5 or 10, and the basic weight lifting coefficient represents how many times the importance of the region of interest is lifted.

[0112] S230, generate a spatial weight map as large as the image using the region of interest mask and the basic weight lifting coefficient , each element of the spatial weight map

[0113] is calculated as follows: ;

[0114] is the value of the pixel position in the region of interest, when the pixel When the pixel is in the region of interest roi, , according to the formula, at this time When the pixel is outside the region of interest roi, , according to the formula, at this time . .

[0115] S240, L1 loss function needs to be calculated pixel by pixel, and the spatial weight map with the same size as the image The spatial dynamic weighted L1 loss function is obtained by weighting the L1 loss function, and the calculation is as follows:

[0116] ;

[0117] The spatial dynamic weighted L1 loss makes the optimizer in the simulator training process pay more attention to the error generated in the region of interest.

[0118] S250, when weighting the multi-scale structural similarity, the region of interest mask is processed in multiple scales to obtain a multi-scale region of interest mask corresponding to the multi-scale image, and the multi-scale region of interest mask is used to give a higher weight to the region of interest at each scale compared with the non-region of interest.

[0119] In the implementation process, the region of interest mask is down-sampled step by step to generate a multi-scale mask pyramid. For example, the average pooling or Gaussian pyramid down-sampling with a step of 2 obtains a multi-scale mask pyramid, which corresponds to a multi-scale image and contains a plurality of multi-scale region of interest masks. The mth scale region of interest mask is:

[0120] , wherein is down-sampling.

[0121] S260, for any one scale region of interest mask , a spatial weight map corresponding to the scale is generated using the basic weight promotion coefficient:

[0122] ;

[0123] , wherein is the pixel coordinate on the scale m image. The spatial weight maps of all scales form a multi-scale spatial weight map.

[0124] S270, the spatial dynamic weighted multi-scale structural similarity loss is calculated using the multi-scale spatial weight map. As described above, the calculation of multi-scale structural similarity involves evaluating contrast and structure at multiple scales respectively. When calculating the structural similarity at any scale m, the spatial weight map of scale m is The weight of the weighted average is provided, and the weighted average contrast and structure indicators are calculated:

[0125]

[0126]

[0127] wherein, represents an image block centered on pixel in the scale m image; is the contrast comparison between image blocks of the scale m image; is the structure comparison between image blocks of the scale m image.

[0128] The weighted average contrast and structure indicators are substituted into the multiscale structure similarity formula between the multiscale image blocks x and y containing the same content to obtain the spatially weighted multiscale structure similarity between the multiscale image blocks x and y containing the same content:

[0129]

[0130] The spatial dynamic weighted multiscale structure similarity between the real image and the simulation image is:

[0131]

[0132] Then, the spatial dynamic weighted multiscale structure similarity loss between the real image and the simulation image is: .

[0133] The spatial dynamic weighted multiscale structure similarity loss enables the optimizer to give higher attention to the image blocks within the region of interest at each scale when calculating the structure similarity, thereby achieving precise spatial dynamic weighting.

[0134] S300, a weight scheduler that dynamically changes with training time is constructed, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function according to the training time during the optimization process, so as to realize the 'from coarse to fine' optimization strategy by using dynamic time scheduling.

[0135] The specific implementation is as follows:

[0136] The total optimization step number of the simulator is set to , and the current is the t-th step, t from 0 to .

[0137] ​​​​A weight scheduler is constructed, and the weight scheduler designs time weights for the base loss and the perception loss respectively and .

[0138] wherein the base loss time weight changes with the training step as follows:

[0139] ;

[0140] decreases linearly from an initial value 1.0 to 0 as the training step increases.

[0141] The perception loss time weight changes with the training step as follows:

[0142] ;

[0143] The perception loss time weight increases linearly from 0 to an ultimate value 1.0 as the training step increases.

[0144] In the early stage of optimization, t is smaller than in the later stage of optimization, is larger than , is smaller than , the base loss contributes more to the initial loss function than the perception loss, and under the action of the base loss, the system focuses on aligning the macro-structure and brightness of the image. In the later stage of optimization, t approaches T max , tends to 0, tends to 1.0, the perception loss contributes more to the initial loss function than the base loss, and under the action of the perception loss, the system turns to finely carve the material, light and shadow and other high-level perceptual details.

[0145] S400, the spatial-dynamic-weighted L1 loss, the spatial-dynamic-weighted multi-scale structural similarity loss, the perception loss and the target loss function calculated by the weight scheduler are integrated to obtain a space-time weighted target loss function, and the target loss function has the following expression form:

[0146] ;

[0147] is the target loss function, and is the final scalar value used by the optimizer to calculate the gradient and update the simulation parameter θ; is the spatial-dynamic-weighted L1 loss; is the spatial-dynamic-weighted multi-scale structural similarity loss.

[0148] S500, calculate the gradient of the simulation parameter of the simulator according to the target loss function, and update the simulation parameter of the simulator according to the gradient iteration using the optimizer, so as to minimize the target loss function.

[0149] There are alternatives for the specific technical modules of the present application, which should also fall within the protection scope of the present application:

[0150] On the optimizer: in addition to the gradient-based optimizer (such as Adam); for more complex black-box simulation models, gradient-independent optimization algorithms such as Bayesian optimization, genetic algorithm or particle swarm optimization algorithm can also be used as alternatives.

[0151] In the measurement of perceptual difference: in addition to using a variational autoencoder, a framework of a generative adversarial network is used to obtain a feature vector representing perception through an encoder of the generative adversarial network, and an adversarial training method is used to measure and narrow the statistical distribution difference between the simulation image and the real image.

[0152] In the simulation modeling: for complex components such as lighting or cameras, in addition to using an analytical model based on physical equations, a data-driven learnable scene representation model can also be used as an alternative, as long as the data-driven learnable scene representation model is parameterized and can be optimized. The data-driven learnable scene representation model includes but is not limited to: a neural radiance field (NeRF) based on continuous function implicit representation and its variants, which use neural network weights to implicitly represent the scene's radiance and density field. A 3D Gaussian splatting model based on discrete primitive explicit representation, which uses a large number of parameterized Gaussian bodies to explicitly construct the scene.

[0153] Embodiment 2

[0154] Referring to Figure 4 As shown in the drawings, the embodiment of the present application provides a simulation-to-reality calibration system based on multi-modal loss and dynamic weighting, which comprises at least one processing unit connected to a storage unit through a bus unit, and the storage unit serves as a computer readable storage medium and can be used to store software programs, computer executable programs and modules, such as the software programs, computer executable programs and modules corresponding to the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting in the embodiment of the present application. The processing unit realizes the above-mentioned simulation-to-reality calibration method based on multi-modal loss and dynamic weighting by running the software programs, computer executable programs and modules stored in the storage unit, which comprises:

[0155] S100, construct a multi-modal and domain-adaptive initial loss function; the initial loss function of the present application comprises a basic loss and a perception loss; the basic loss comprises an L1 loss and a multi-scale structural similarity loss;

[0156] S200, generating a spatial weight map for spatially dynamically weighting the constructed multi-modal, domain-adaptive initial loss function according to the positional relationship between the pixel regions of interest;

[0157] S300, constructing a weight scheduler that dynamically changes with training time, the weight scheduler being responsible for dynamically adjusting the weights of different loss terms in the initial loss function according to training time during optimization;

[0158] S400, calculating a spatio-temporally weighted target loss function by comprehensively combining the spatial weight map, the weight scheduler, and the initial loss function;

[0159] S500, calculating the gradient of the simulation parameters of the simulator according to the target loss function, and updating the simulator parameters iteratively using an optimizer according to the gradient, so as to minimize the target loss function.

[0160] Of course, the storage unit in the simulation-to-reality calibration system based on multi-modal loss and dynamic weighting provided by the embodiments of the present application stores a computer program that is not limited to the method operations described above, but can also perform related operations in the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting provided by any embodiment of the present application.

[0161] Embodiment 3

[0162] The embodiments of the present application provide a computer readable storage medium storing a computer program, the computer program being executed to implement the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting, comprising:

[0163] S100, constructing a multi-modal, domain-adaptive initial loss function; the initial loss function of the present application includes a basic loss and a perception loss; the basic loss includes an L1 loss and a multi-scale structural similarity loss;

[0164] S200, generating a spatial weight map for spatially dynamically weighting the constructed multi-modal, domain-adaptive initial loss function according to the positional relationship between the pixel regions of interest;

[0165] S300, constructing a weight scheduler that dynamically changes with training time, the weight scheduler being responsible for dynamically adjusting the weights of different loss terms in the initial loss function according to training time during optimization;

[0166] S400, calculating a spatio-temporally weighted target loss function by comprehensively combining the spatial weight map, the weight scheduler, and the initial loss function;

[0167] S500, calculate the gradient of the simulation parameter of the simulator according to the target loss function, and update the simulation parameter of the simulator according to the gradient iteratively using the optimizer, so as to minimize the target loss function.

[0168] The computer program stored in the computer readable storage medium provided by the embodiments of the present application is not limited to the method operations described above, and can also perform related operations in the simulation-to-reality calibration method based on multi-modal loss and dynamic weighting provided by any embodiment of the present application.

[0169] In the embodiments provided by the present application, it should be understood that the disclosed structure and method can be implemented in other ways. For example, the above-described structural embodiments are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, structures or units, and can be electrical, mechanical or in other forms.

[0170] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0171] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of software functional unit.

[0172] The above is only a specific implementation of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A simulation-to-real calibration method based on multi-modal loss with dynamic weighting, characterized in that, Comprise: S100, construct a multi-modal, domain adaptive initial loss function for training the simulator; the initial loss function contains a basic loss and a perception loss; the basic loss contains an L1 loss and a multi-scale structural similarity loss; S200, generate a spatial weight map and a multi-scale spatial weight map for spatial dynamic weighting of the constructed multi-modal, domain adaptive initial loss function according to the positional relationship between the pixel regions of interest; the spatial weight map and the multi-scale spatial weight map respectively weight the L1 loss and the multi-scale structural similarity loss; S300, construct a weight scheduler that dynamically changes with training time, which is responsible for dynamically adjusting the weights of different loss terms in the initial loss function during the optimization process of the simulator according to the training time; S400, integrate the spatially dynamically weighted L1 loss, the spatially dynamically weighted multi-scale structural similarity loss, the perception loss, and the spatio-temporally weighted target loss function calculated by the weight scheduler; S500, calculate the gradient of the simulation parameters of the simulator according to the target loss function, and use the optimizer to iteratively update the simulator parameters according to the gradient to minimize the target loss function.

2. The multi-modal loss and dynamic weighting based simulation-to- reality calibration method of claim 1, wherein, The process of constructing the initial loss function is as follows: S110, calculating the simulation image L1 loss and multi-scale structural similarity loss between the corresponding real image and the basis loss; S120, use a pre-trained variational autoencoder to construct a self-supervised perception loss.

3. The multi-modal loss and dynamic weighting based simulation-to- reality calibration method of claim 2, wherein, Using a pre-trained variational autoencoder to construct a self-supervised perception loss includes: Use an unlabeled real image to pre-train a variational autoencoder, which consists of an encoder and a decoder; After pre-training, obtain the encoder of the variational autoencoder, and freeze the encoder weight of the variational autoencoder as a domain expert encoder; In the simulation-to-reality calibration process, the domain expert encoder acts as a fixed 'feature extractor' to convert the input simulation and real images into feature vectors in the latent space; The distance between the feature vectors in the latent space is taken as the perception loss between the simulation image and the real image.

4. The multi-modal loss and dynamic weighting based simulation-to- reality calibration method of claim 3, wherein, When training the variational autoencoder, the real image is input into the encoder of the variational autoencoder to generate a mean vector and a log variance vector representing the distribution of the feature vectors in the latent space; then sample the feature vectors in the latent space from the mean vector and the log variance vector, and input them into the decoder of the variational autoencoder to reconstruct the original real image; The goal of training the variational autoencoder is to minimize the reconstruction loss and the KL divergence, where the reconstruction loss is the difference between the original real image and the reconstructed image, and the KL divergence is the difference between the distribution generated by the encoder and the standard normal distribution.

5. The multi-modal loss with dynamic weighting based simulation-to- reality calibration method of claim 1, wherein, S200 includes: S210, generating a binary region of interest mask by a segmentation model or prior knowledge , the region of interest mask has the same size as the image; S220, defining a base weight boosting coefficient , base weight boosting coefficient , the base weight boosting coefficient represents how many times the importance of the region of interest is boosted; S230, generating a spatial weight map of the same size as the image using the region of interest mask and the base weight boost coefficients , each element of the spatial weight map is computed as follows: ； For a pixel position within the region of interest, when the pixel Within the region of interest roi, According to the formula, at this time ; when the pixel Outside the region of interest roi, According to the formula, at this time ; S240, a spatial weight map of the same size as the image The weighted L1 loss function obtains a spatial dynamic weighted L1 loss. S250, perform multi-scale processing on the region of interest mask to obtain a multi-scale region of interest mask corresponding to the multi-scale image; S260, for any one scale of interest region mask, a corresponding scale of spatial weight map is generated by using the base weight lifting coefficient : ； wherein, are pixel coordinates on the scale m image, spatial weight maps for all scales forming multi-scale spatial weight maps; S270, calculate the spatially dynamically weighted multi-scale structural similarity loss using the multi-scale spatial weight map.

6. The multi-modal loss with dynamic weighting based simulation-to- reality calibration method of claim 5, wherein, The spatial weight map The spatial dynamic weighted L1 loss function is calculated as follows: ； wherein, spatially dynamically weighted L1 loss causes the optimizer of the simulator training process to focus on errors produced in the region of interest.

7. The multi-modal loss with dynamic weighting based simulation-to- reality calibration method of claim 5, wherein, Calculating the spatially dynamically weighted multi-scale structural similarity loss using the multi-scale spatial weight map includes: A scale m spatial weight map The weight of the weighted average is provided, and the weighted average contrast and texture index are calculated: ； ； wherein, represents an image block centered on a pixel on a scale m image; is a contrast comparison between image blocks of a scale m image; is a structure comparison between image blocks of a scale m image; m = 1, 2,... M; M is the number of scales. Substitute the weighted average contrast and structure index into the multi-scale structural similarity formula between the multi-scale image blocks x and y containing the same content to obtain the spatially weighted multi-scale structural similarity between the multi-scale image blocks x and y containing the same content: ； Real image and simulated image spatially dynamically weighted multi-scale structural similarity between: ； Then, the spatial dynamic weighted multi-scale structural similarity loss between the real image and the simulated image is: ​ 8. The multi-modal loss with dynamic weighting based simulation-to- reality calibration method of claim 1, wherein, The weight scheduler is constructed in the following manner, comprising: S310, set the total optimization step number of the simulator as , the current is the t step, t from 0 to ; S320, constructing a weight scheduler, the weight scheduler is designed for a time weight of the basic loss and the perception loss respectively and ; where the base loss time weight Changes with training steps are as follows: ； linearly decreases from an initial value 1.0 to 0 as the training step increases; Perceptual loss time weight Varies with training step as follows: ； Perceptual loss time weight Linearly rises from 0 to a final value of 1.0 as the training step increases. In the early stage of optimization, Compared with Larger, the contribution of the base loss to the initial loss function is larger than that of the perception loss, and the macro structure and brightness of the aligned image are aligned under the action of the base loss; in the later stage of optimization, t gradually approaches T max , With the increase of t, it gradually tends to 0, Tends to 1.0, the contribution of the perception loss to the initial loss function is larger than that of the base loss, and the perception details are modeled under the action of the perception loss.

9. A simulation-to-reality calibration device based on multi-modal loss with dynamic weighting, comprising: At least one processing unit, the processing unit is connected with the storage unit through the bus unit, characterized in that the storage unit stores the computer program, and the processing unit realizes the simulation-to-reality calibration method based on the multi-modal loss and dynamic weighting according to any one of claims 1-8 by running the computer program stored in the storage unit.

10. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed to realize the simulation-to-reality calibration method based on the multi-modal loss and dynamic weighting according to any one of claims 1-8.

Images