Image enhancement method based on causal reverse reconstruction and physical consistency constraint

Abstract

The application provides an image enhancement method based on causal reverse reconstruction and physical consistency constraint, comprising: reverse reconstructing a causal parameter vector from an input image, and calculating an optimal causal parameter vector according to the causal parameter vector and a target function, wherein the causal parameter vector comprises scene geometry, material attribute, illumination parameter and camera parameter, and the target function comprises a pixel matching term, a gradient domain consistency term, a physical constraint term and a regularization term; applying a parameter perturbation term to the optimal causal parameter vector to generate an enhanced causal parameter vector, wherein the amplitude of the parameter perturbation term is adaptively set according to category scarcity; generating a candidate enhanced image and a candidate multi-modal product by using a differentiable renderer according to the enhanced causal parameter vector; performing physical consistency verification on the candidate enhanced image and the candidate multi-modal product, and outputting the enhanced image and the multi-modal product that pass the verification based on the verification result. The method of the application can enhance the reality of the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an image enhancement method and device based on causal reverse reconstruction and physical consistency constraint. BACKGROUND

[0002] In many application fields of computer vision, graphics and artificial intelligence, it is often necessary to obtain a large number of images with high physical realism and diversity. For example, when training a deep learning model, high-quality image data is needed to improve the generalization ability of the model; in visual effects, virtual reality or autonomous driving simulation, realistic images that conform to physical laws are also needed.

[0003] At present, the method of generating new images mainly includes two categories. One is the traditional image enhancement method, such as image rotation, cropping, color jittering, etc. This kind of method makes simple transformation in the pixel space, lacks modeling of the imaging causal chain (such as scene geometry, material, lighting), and often violates the physical law in shadow, occlusion, perspective, etc., resulting in insufficient realism and limited diversity.

[0004] The other category is the method based on generative model (such as generative adversarial network GAN, diffusion model) or neural rendering. Although these methods can generate images with good visual effects, they are mostly limited to the surface level of appearance and fail to decouple and manipulate the underlying physical parameters of the scene. Therefore, the controllability of the generated results is poor, it is difficult to ensure its physical rationality, and it is also difficult to conveniently generate multi-modal data (such as depth map, normal map) that perfectly match the image.

[0005] Therefore, there is a lack of an image enhancement method and device in the prior art that can start from a single image, decouple and manipulate its internal physical causal parameters, and then generate images that are highly realistic, diverse and strictly conform to physical laws. SUMMARY

[0006] Therefore, it is necessary to provide an image enhancement method and device based on causal reverse reconstruction and physical consistency constraint in view of the above technical problems.

[0007] In a first aspect, the present application provides an image enhancement method based on causal reverse reconstruction and physical consistency constraint, comprising:

[0008] reversely reconstructing a causal parameter vector from an input image, and calculating an optimal causal parameter vector according to the causal parameter vector and a target function, wherein the causal parameter vector includes scene geometry parameters, material attribute parameters, lighting parameters and camera parameters, and the target function includes a pixel matching term, a gradient domain consistency term, a physical constraint term and a regularization term;

[0009] A parameter perturbation term is imposed on the optimal causal parameter vector to generate an enhanced causal parameter vector, wherein the amplitude of the parameter perturbation term is adaptively set according to the category scarcity;

[0010] A differentiable renderer is used to generate a candidate enhanced image and a candidate multi-modal product according to the enhanced causal parameter vector;

[0011] The candidate enhanced image and the candidate multi-modal product are subjected to physical consistency checking, and based on the checking result, the enhanced image and the multi-modal product that pass the checking are output.

[0012] Optionally, after the candidate enhanced image and the candidate multi-modal product are subjected to physical consistency checking, and based on the checking result, the enhanced image and the multi-modal product that pass the checking are output, the method further comprises:

[0013] A two-dimensional label of the enhanced image is generated by perspective projection and differentiable rasterization according to the enhanced causal parameter vector corresponding to the enhanced image and the multi-modal product that pass the checking;

[0014] An enhanced sample is generated according to the enhanced image that passes the checking, the multi-modal product and the two-dimensional label, wherein the enhanced sample comprises the enhanced image, the two-dimensional label, the multi-modal product and related metadata.

[0015] Optionally, the candidate enhanced image and the candidate multi-modal product are subjected to physical consistency checking, and based on the checking result, the enhanced image and the multi-modal product that pass the checking are output, comprising:

[0016] The candidate enhanced image and the candidate multi-modal product are subjected to physical consistency checking;

[0017] If the physical consistency loss is less than a preset threshold, the candidate enhanced image and the candidate multi-modal product are taken as the enhanced image and the multi-modal product that pass the checking;

[0018] If the physical consistency loss is greater than or equal to the preset threshold, the enhanced causal parameter vector corresponding to the candidate enhanced image and the candidate multi-modal product is modified, and the following rendering and checking steps are repeatedly executed until a output condition is met:

[0019] A differentiable renderer is used to generate a candidate enhanced image and a candidate multi-modal product according to the modified enhanced causal parameter vector, and a physical consistency loss is calculated according to the candidate enhanced image and the candidate multi-modal product, and the candidate enhanced image and the candidate multi-modal product are subjected to physical consistency checking;

[0020] If the physical consistency loss is greater than or equal to the preset threshold, the enhanced causal parameter vector corresponding to the candidate enhanced image and the candidate multi-modal product is modified, and the cycle continues;

[0021] If the physical consistency loss is less than a preset threshold, the loop is exited and the candidate enhanced image and candidate multimodal product generated this time are taken as the enhanced image and multimodal product that pass the verification.

[0022] Optionally, the causal parameter vector is reconstructed inversely from the input image, and the optimal causal parameter vector is calculated based on the causal parameter vector and the objective function, including:

[0023] Obtain the initial causal parameter vector from the input image;

[0024] Repeat the following steps:

[0025] Input the current causal parameter vector into the differentiable renderer to obtain the corresponding rendered image;

[0026] Calculate the pixel-level differences between the input image and the rendered image, and calculate pixel matching terms based on the pixel-level differences between the input image and the rendered image;

[0027] Calculate the gradient map difference between the input image and the rendered image, and calculate the gradient domain consistency term based on the gradient map difference between the input image and the rendered image and the preset gradient domain weights;

[0028] Based on physical consistency constraints, calculate the physical consistency penalty of the current causal parameter vector, and calculate the physical constraint terms based on the physical consistency penalty and the preset physical constraint weights.

[0029] Calculate the difference between the current causal parameter vector and the prior causal parameter vector, and calculate the regularization term based on the difference between the current causal parameter vector and the prior causal parameter vector and the preset regularization weight.

[0030] The objective function is obtained by adding the pixel matching term, gradient domain consistency term, physical constraint term, and regularization term.

[0031] Determine whether the sequence of all objective functions meets the preset convergence conditions;

[0032] If the sequence of all objective functions does not meet the preset convergence condition, update the current causal parameter vector based on the gradient of the objective function with respect to the current causal parameter vector, and continue the loop.

[0033] If the sequence of all objective functions meets the preset convergence condition, exit the loop and take the causal parameter vector that minimizes the objective function in the sequence of all objective functions as the optimal causal parameter vector.

[0034] Alternatively, the optimal causal parameter vector can be calculated using the following formula:

[0035]

[0036] in, is an optimal causal parameter vector; is a pixel matching term, representing the input image and the rendered image in terms of pixel-level difference; is a gradient domain weight; is an input image and a rendered image in terms of gradient map difference; is a gradient domain consistency term; is a physical constraint weight; is a physical consistency penalty; is a physical constraint term; is a regularization weight; is a current causal parameter vector and a prior causal parameter vector in terms of L2 norm of difference; is a regularization term; denotes an objective function; denotes taking the causal parameter vector that minimizes the objective function in a sequence of all objective functions.

[0037] Optionally, a parameter perturbation term is imposed on the optimal causal parameter vector to generate an enhanced causal parameter vector, including:

[0038] The number of samples of each category in the statistical training data set is counted, and the sample proportion of each category is calculated;

[0039] According to the sample proportion, the scarcity weight of each category is calculated, wherein the scarcity weight is negatively correlated with the sample proportion;

[0040] According to the scarcity weight, the perturbation amplitude range of each category is determined;

[0041] Random perturbations are imposed on the scene geometry parameters, material attribute parameters, lighting parameters and camera parameters in the optimal causal parameter vector, and the optimal causal parameter vector after the perturbations are imposed is taken as the enhanced causal parameter vector.

[0042] Optionally, according to the enhanced causal parameter vector corresponding to the enhanced image and the multi-modal product that pass the verification, a two-dimensional label of the enhanced image is generated through perspective projection and differentiable rasterization, including:

[0043] According to the scene geometry parameters in the enhanced causal parameter vector, the three-dimensional bounding box vertex coordinates of each object in the scene are obtained;

[0044] According to the intrinsic parameters and extrinsic parameters in the camera parameters in the enhanced causal parameter vector, a perspective projection matrix is constructed;

[0045] transforming the three-dimensional bounding box vertex of each object to the two-dimensional image space through a perspective projection matrix;

[0046] calculating the minimum circumscribed rectangle of each object projected in the two-dimensional image space to obtain the corresponding two-dimensional bounding box label;

[0047] According to the scene geometry information in the enhanced causal parameter vector, the segmentation mask of each object is generated through differentiable rasterization;

[0048] obtaining the pre-defined three-dimensional key point coordinates of each object, and transforming the three-dimensional key point coordinates to the two-dimensional image space through a perspective projection matrix to obtain the corresponding two-dimensional key point coordinates;

[0049] combining the two-dimensional bounding box, the segmentation mask and the two-dimensional key point coordinates to form the two-dimensional label of the enhanced image.

[0050] Optionally, the candidate enhanced image and the candidate multi-modal product are subjected to physical consistency verification, including:

[0051] verifying whether the material reflection characteristic in the material attribute parameter in the enhanced causal parameter vector conforms to the law of conservation of energy;

[0052] If the material reflection characteristic does not conform to the law of conservation of energy, it is determined that the physical consistency loss is greater than or equal to a preset threshold, and the current verification is ended;

[0053] If the material reflection characteristic conforms to the law of conservation of energy, the following verification steps are continued to be executed:

[0054] obtaining a shadow consistency loss by comparing the edge alignment degree and the intensity conservation between the candidate enhanced image and the shading map rendered by the enhanced causal parameter vector;

[0055] obtaining an occlusion order loss by verifying whether the foreground object depth is less than the background object depth through analyzing the depth map and the occlusion map in the candidate multi-modal product;

[0056] combining the shadow consistency loss and the occlusion order loss according to the weight to obtain the physical consistency loss.

[0057] Optionally, when processing a video sequence, the method further comprises:

[0058] applying a smooth prior constraint to the perturbation of the enhanced causal parameter vector between adjacent frames, wherein, from the second frame, the perturbation of the t-th frame t is generated based on the perturbation of the t-1-th frame (t-1) , and satisfies and ;

[0059] wherein, is the perturbation of the t-th frame. is the disturbance of the t-1th frame; is the disturbance change amount; is the disturbance is the covariance between each component in the disturbance; is the smoothing threshold; represents the disturbance of the tth frame , which is randomly sampled from a Gaussian distribution (i.e., normal distribution) with a mean of and a covariance of .

[0060] In a second aspect, the present application provides an image enhancement device based on causal inverse reconstruction and physical consistency constraint, comprising:

[0061] A parameter reconstruction module is configured to inversely reconstruct a causal parameter vector from an input image, and calculate an optimal causal parameter vector according to the causal parameter vector and a target function, wherein the causal parameter vector includes scene geometry, material attribute, lighting parameter and camera parameter, and the target function includes a pixel matching term, a gradient domain consistency term, a physical constraint term and a regularization term;

[0062] A parameter enhancement module is connected to the parameter reconstruction module, configured to apply a parameter disturbance term to the optimal causal parameter vector to generate an enhanced causal parameter vector, wherein the amplitude of the parameter disturbance term is adaptively set according to the category scarcity;

[0063] A rendering module is connected to the parameter enhancement module, configured to generate a candidate enhanced image and a candidate multi-modal product by using a differentiable renderer according to the enhanced causal parameter vector;

[0064] A verification module is connected to the rendering module, configured to perform physical consistency verification on the candidate enhanced image and the candidate multi-modal product, and output the enhanced image and the multi-modal product that pass the verification based on the verification result.

[0065] The image enhancement method and device based on causal inverse reconstruction and physical consistency constraint of the present application firstly generates enhanced samples with higher realism and physical rationality in terms of lighting, shadow, perspective, etc. by enhancing in the physical parameter space instead of traditional pixel transformation, effectively avoiding geometric distortion and optical abnormalities. Secondly, based on the automatic label reconstruction mechanism of rendering geometry, the boundary box, segmentation mask and key point label accurately registered with the enhanced image can be generated synchronously, completely eliminating the annotation error accumulation problem caused by approximate transformation in traditional enhancement methods, and realizing a zero-annotation-error automatic process. In addition, this method naturally supports consistency enhancement of multi-modal data and video sequences, and guarantees inter-frame continuity through temporal smoothing constraint, providing high-quality training data for video understanding tasks. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1a A flowchart of the image enhancement method based on the causal reverse reconstruction and the physical consistency constraint provided by an embodiment of the present application is shown in FIG. 1.

[0067] Figure 1b Another flowchart of the image enhancement method based on the causal reverse reconstruction and the physical consistency constraint provided by an embodiment of the present application is shown in FIG. 2.

[0068] Figure 1c Still another flowchart of the image enhancement method based on the causal reverse reconstruction and the physical consistency constraint provided by an embodiment of the present application is shown in FIG. 3.

[0069] Figure 2 A circuit module structure diagram of the image enhancement device based on the causal reverse reconstruction and the physical consistency constraint provided by an embodiment of the present application is shown in FIG. 4.

[0070] Figure 3 An internal structure diagram of the computer device in an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0071] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0072] As shown in FIG. 1, the present application provides an image enhancement method based on causal reverse reconstruction and physical consistency constraint, which comprises: Figure 1a

[0073] Step S11: reverse reconstructing a causal parameter vector from an input image, and calculating an optimal causal parameter vector according to the causal parameter vector and a target function, wherein the causal parameter vector comprises scene geometry parameters, material attribute parameters, illumination parameters and camera parameters, and the target function comprises a pixel matching term, a gradient domain consistency term, a physical constraint term and a regularization term.

[0074] In the present application, for the causal parameter vector, a person skilled in the art can flexibly select according to actual needs, such as further including weather or medium scattering parameters, which are not limited herein.

[0075] The scene geometry parameters comprise three-dimensional points, meshes, volume representations, etc.; the material attribute parameters comprise BRDF (Bidirectional Reflectance Distribution Function or reflectivity model or material model) parameters (such as roughness a, base reflectance F0, etc.); the illumination parameters are directional light parameters, ambient light parameters, area light parameters, etc.; and the camera parameters comprise intrinsic parameters (intrinsic matrix), extrinsic parameters (rotation matrix and translation vector in extrinsic matrix), etc.

[0076] ​Optionally, the step of inversely reconstructing the causal parameter vector from the input image in the step S11, and calculating the optimal causal parameter vector according to the causal parameter vector and the objective function, specifically comprises:

[0077] Step S110: obtaining an initial causal parameter vector from the input image;

[0078] In the present application, the initial causal parameter vector can be obtained from the input image in combination with the monocular depth estimation algorithm, the surface normal estimation algorithm, the illumination estimation algorithm and the intrinsic image decomposition algorithm in the prior art. Of course, the initial causal parameter vector can also be obtained by selecting other methods according to actual needs, which is not limited herein.

[0079] In an optional embodiment of the present application, the step S110 specifically comprises:

[0080] Step S1101: performing three-dimensional structure analysis on the input image based on the monocular depth estimation algorithm to obtain the depth information and the three-dimensional geometric layout of the scene, and obtaining the scene geometric parameters;

[0081] Step S1102: performing surface orientation analysis on the input image based on the surface normal estimation algorithm to calculate the normal direction of each pixel point, and perfecting the surface detail information of the scene geometry;

[0082] Step S1103: performing material and illumination separation processing on the input image based on the intrinsic image decomposition algorithm to obtain the albedo map and the material reflection characteristics, and obtaining the material attribute parameters;

[0083] Step S1104: performing shadow and highlight analysis on the input image based on the illumination estimation algorithm to infer the light source direction, intensity and color characteristics, and obtaining the illumination parameters;

[0084] Step S1105: estimating the intrinsic and extrinsic parameters of the input image based on the camera calibration algorithm, establishing the projection relationship from the three-dimensional space to the two-dimensional image, and initializing the camera parameters.

[0085] Specifically, if the scene geometric parameters are G, the material attribute parameters are M, the illumination parameters are L, and the camera parameters are C=(K, R, t), the initial causal parameter vector is {G, M, L, C}. Among them, K in the camera parameter C is the intrinsic matrix, that is, the intrinsic parameter, which contains the focal length and the principal point parameter; R is the rotation matrix, and t is the translation vector, which together constitute the extrinsic matrix, that is, the extrinsic parameter. If the causal parameter vector further includes weather or medium scattering parameters X, the initial causal parameter vector is {G, M, L, C, X}.

[0086] After the step S110, the following steps are executed in a loop:

[0087] Step S111: input the current causal parameter vector into the differentiable renderer to obtain a corresponding rendered image;

[0088] It should be noted that the current causal parameter vector is a dynamically changing parameter vector. For example: after obtaining the initial causal parameter vector from the input image, the initial causal parameter vector is input into the differentiable renderer as the current causal parameter vector to obtain a corresponding rendered image; and after a cycle, if the sequence of all objective functions does not meet the preset convergence condition, the current causal parameter vector is updated based on the gradient of the objective function on the current causal parameter vector, and a new cycle is continued, then the current causal parameter vector input into the differentiable renderer is the updated current causal parameter vector, and so on, which will not be repeated here.

[0089] Among them, the differentiable renderer adopts the existing differentiable renderer, such as the PyRedner differentiable renderer, and the person skilled in the art can flexibly select according to actual needs, which is not limited here.

[0090] Step S112: calculate the pixel-level difference between the input image and the rendered image, and based on the pixel-level difference between the input image and the rendered image, calculate the pixel matching term;

[0091] Step S113: calculate the gradient map difference between the input image and the rendered image, and based on the gradient map difference between the input image and the rendered image and the preset gradient domain weight, calculate the gradient domain consistency term;

[0092] Step S114: based on the physical consistency constraint, calculate the physical consistency penalty of the current causal parameter vector, and based on the physical consistency penalty and the preset physical constraint weight, calculate the physical constraint term;

[0093] Step S115: calculate the difference between the current causal parameter vector and the prior causal parameter vector, and based on the difference between the current causal parameter vector and the prior causal parameter vector and the preset regularization weight, calculate the regularization term;

[0094] Step S116: add the pixel matching term, the gradient domain consistency term, the physical constraint term and the regularization term to obtain the objective function;

[0095] Step S117: determine whether the sequence of all objective functions meets the preset convergence condition;

[0096] In the present application, the preset convergence condition can be flexibly set by the person skilled in the art according to actual needs, which is not limited here. In a specific embodiment, the preset convergence condition is that the difference between adjacent objective function values is less than a preset difference threshold in continuous multiple iterations (circulations); for example, the variation of the objective function value is less than the preset difference threshold in 20 consecutive iterations. The number of iterations and the preset difference threshold can be flexibly set by the person skilled in the art according to actual needs, which is not limited here.

[0097] Step S118: If the sequence composed of all the objective functions does not meet the preset convergence condition, the current causal parameter vector is updated based on the gradient of the objective function on the current causal parameter vector, and the steps S111 to S117 are continued to be circulated;

[0098] Step S119: If the sequence composed of all the objective functions meets the preset convergence condition, the circulation is exited, and the causal parameter vector that makes the objective function in the sequence composed of all the objective functions minimum is taken as the optimal causal parameter vector.

[0099] The optimal causal parameter vector can be calculated by the following formula:

[0100]

[0101] is the optimal causal parameter vector; is a pixel matching term, representing the L1 norm of the pixel-level difference between the input image and the rendered image ; is a gradient domain weight; is the L1 norm of the gradient map difference between the input image and the rendered image ; is a gradient domain consistency term; is a physical constraint weight; is a physical consistency penalty; is a physical constraint term; is a regularization weight; is the L2 norm of the difference between the current causal parameter vector and the prior causal parameter vector ; is a regularization term; represents the objective function; represents the causal parameter vector that makes the objective function in the sequence composed of all the objective functions minimum.

[0102] The gradient domain weight , the physical constraint weight , and the regularization weight ​The technical personnel in the art can flexibly set according to actual needs, which is not limited here.

[0103] Step S12: applying a parameter disturbance term to the optimal causal parameter vector to generate an enhanced causal parameter vector, wherein the amplitude of the parameter disturbance term is adaptively set according to the category scarcity degree;

[0104] Optionally, as shown in Figure 1b , the step S12 of applying a parameter disturbance term to the optimal causal parameter vector to generate an enhanced causal parameter vector specifically includes:

[0105] Step S121: counting the number of samples of each category in the training data set, and calculating the sample proportion of each category;

[0106] Step S122: calculating the scarcity weight of each category according to the sample proportion, wherein the scarcity weight is negatively correlated with the sample proportion;

[0107] Step S123: determining the disturbance amplitude range of each category according to the scarcity weight;

[0108] Step S124: applying random disturbance to the scene geometry parameters, material attribute parameters, lighting parameters and camera parameters in the optimal causal parameter vector, and taking the optimal causal parameter vector after disturbance as the enhanced causal parameter vector.

[0109] In the present application, the scarcity weight of each category can be calculated by the following formula: , wherein, is the category; is the scarcity weight of the category is the sample proportion of the category ; is the compression index, .

[0110] In the present application, the enhanced causal parameter vector can be calculated by the following formula: , wherein, is the enhanced causal parameter vector; is the optimal causal parameter vector; is the disturbance; is a probability distribution family parameterized by the scarcity weight and the covariance matrix , such as Gaussian distribution, uniform distribution, etc.; is the scarcity weight; is the covariance between the components in the disturbance ; indicates that the enhanced causal parameter vector is generated by sampling from the probability distribution family parameterized by the scarcity weight and covariance matrix Commonly parameterized probability distribution family In the specific implementation, the random sampling obtains the perturbation .

[0111] For parameters (such as angles and positions) that are equally reasonable for values within a limited range, a uniform distribution is used to fully explore the parameter space; for continuous parameters (such as intensity and material properties) that tend to have small amplitude changes, a Gaussian distribution is used to maintain the authenticity and naturalness of the samples.

[0112] Specifically, if the optimal causal parameter vector , , And , , According to the scarcity weight and the corresponding covariance matrix The commonly parameterized probability distribution family randomly selects a perturbation , and then adjusts the optimal causal parameter vector to obtain an enhanced causal parameter vector. It should be noted that the specific embodiment omits the camera parameter C for the convenience of description.

[0113] Step S13: generating a candidate enhanced image and a candidate multi-modal product using a differentiable renderer according to the enhanced causal parameter vector;

[0114] The multi-modal product includes a depth map, a normal map, an occlusion map, an albedo map, point cloud data, a segmentation map, an instance ID, and the like. The multi-modal product can be generated flexibly according to the actual needs of those skilled in the art, and is not limited herein.

[0115] Step S14: performing physical consistency verification on the candidate enhanced image and the candidate multi-modal product, and outputting the enhanced image and the multi-modal product that pass the verification based on the verification result.

[0116] Optionally, step S14 specifically includes:

[0117] Step S141: performing physical consistency verification on the candidate enhanced image and the candidate multi-modal product in step S13;

[0118] In the present application, the physical consistency verification on the candidate enhanced image and the candidate multi-modal product in step S141 can specifically include:

[0119] Step S1411: verifying whether the material reflection characteristic in the material property parameter in the enhanced causal parameter vector satisfies the law of conservation of energy;

[0120] Optionally, whether the material reflection characteristic in the enhanced causal parameter vector satisfies the law of conservation of energy is calculated using the following formula: ,in, The direction of the incident light; The direction of launch or the direction of observation; The angle between the incident light and the surface normal; The term is the cosine of the angle of incidence; For differential solid angles; It is a two-way reflection distribution function; This indicates that the integral is performed over all incident directions in the entire upper hemisphere space with the surface normal as the axis.

[0121] Step S1412: If the material's reflection characteristics do not conform to the law of conservation of energy, determine that the physical consistency loss is greater than or equal to a preset threshold, and end the current verification.

[0122] Step S1413: If the material's reflection properties conform to the law of conservation of energy, continue with the following verification steps:

[0123] Step S1414: Obtain the shadow consistency loss by comparing the edge alignment and intensity conservation between the candidate enhanced image and the shading map rendered by the enhancement causal parameter vector;

[0124] The shadow consistency loss in step S1414 can be calculated using the following formula: ,in, This represents the loss of shadow consistency. Candidate enhanced images; To enhance candidate images The edge map extracted from it; To extract from the color map The edge map extracted from it; For edge alignment, this represents the reciprocal of the intersection-union ratio (CIU) of two edge maps. For strength conservation weights; Indicating in candidate enhancement images Within the shaded area, sum the brightness values ​​of all pixels; In the coloring diagram Within the shaded area, sum the brightness values ​​of all pixels; This indicates that the absolute value of the difference between the two sums is calculated. This is a strength conservation term.

[0125] Among them, the coloring image Differentiable renderers can be used directly to generate candidate enhancement images. Time-synchronized output shader Alternatively, a theoretical shading map can be recalculated based on scene geometry and lighting parameters using the existing Lambertian model. , used for calculating physical consistency loss.

[0126] For the candidate enhanced image and the shading map , the Canny algorithm, Sobel algorithm, etc. in the prior art can be used for edge extraction, which is not limited here.

[0127] Optionally, is calculated by the following method:

[0128] ① Create a shadow area mask: set a brightness threshold (for example, threshold = 0.3, assuming the shading map value range is [0, 1]) for the theoretical shading map Ŝ(θ'); create a binary mask , where: = 1 (True) if < threshold (the pixel is in the shadow), = 0 (False) if >= threshold (the pixel is not in the shadow), is the pixel.

[0129] ② Extract the shadow area pixels: convert the candidate enhanced image (which is usually an RGB color image) into a grayscale image ; use the shadow mask created in the previous step to extract the corresponding area in : , where, represents the Hadamard product. Usually only the corresponding pixels with mask value 1 are retained, and other positions are set to 0 or ignored, in other words, the mask value 1 indicates that the pixel is in the shadow area.

[0130] , the calculation method of is the same as that of , which is not repeated here.

[0131] Step S1415: Verify whether the foreground object depth is less than the background object depth by analyzing the depth map and the occlusion map in the candidate multi-modal product, to obtain an occlusion order loss;

[0132] The occlusion order loss in step S1415 can be calculated by the following formula: , where, is the occlusion order loss; is the occlusion indication value at pixel ; is the depth value of the foreground object at pixel ; is the depth value of the background object at pixel ; As a safety margin, used to tolerate numerical errors in depth calculations, its value ranges from [1e-5, 1e-2] (normalized value) to [0.001, 0.01] (actual physical value). This is a penalty item.

[0133] It should be noted that, This indicates traversing all pixels in the candidate enhancement image. When in pixels When the occlusion indicator value at a certain location is 1, the condition is considered true, and a return value of 1 is returned; when the occlusion indicator value at a certain location is 1, the condition is considered true, and a return value of 1 is returned. When the occlusion indicator value is 0, it is judged as false and a value of 0 is returned; after the judgment is completed, the value of each pixel is calculated. The sum of the return values. Furthermore, It should be smaller than (The foreground is closer to the camera), at this time If the number is negative, adding δ may still result in a negative number. It will take 0, indicating no penalty; if Greater than (The foreground is farther from the camera), which violates the laws of physics. At this point, If the number is positive, adding δ will result in a positive number. The value will be positive, and its magnitude reflects the severity of the violation.

[0134] In this invention, at the pixel Obstruction indicator value at the location Obtained from the occlusion map in the candidate multimodal products; in pixels Depth value of foreground object and in pixels Depth value of the background object It is obtained from the depth map in the candidate multimodal products.

[0135] Step S1416: Combine the shadow consistency loss and occlusion ordering loss according to their weights to obtain the physical consistency loss.

[0136] Optionally, physical consistency loss ,in, Weights for shadow consistency loss; Weights are lost to mask the orderliness.

[0137] Among them, the weight of shadow consistency loss And occlusion order loss weight Those skilled in the art can set it flexibly according to actual needs, and no limitation is made here.

[0138] It should be noted that physical consistency loss That is .

[0139] Step S142: If the physical consistency loss is less than the preset threshold, the candidate enhanced image and the candidate multi-modal product are taken as the enhanced image and the multi-modal product that pass the verification.

[0140] Step S143: If the physical consistency loss is greater than or equal to the preset threshold, the enhanced causal parameter vector corresponding to the candidate enhanced image and the candidate multi-modal product in step S13 is modified, and the following rendering and verification steps are executed in a loop until the output condition is met:

[0141] Step S144: According to the modified enhanced causal parameter vector, the candidate enhanced image and the candidate multi-modal product are generated by using the differentiable renderer, and the physical consistency loss is calculated according to the candidate enhanced image and the candidate multi-modal product, and the physical consistency of the generated candidate enhanced image and the candidate multi-modal product is verified.

[0142] Step S145: If the physical consistency loss is greater than or equal to the preset threshold, the enhanced causal parameter vector corresponding to the candidate enhanced image and the candidate multi-modal product is modified, and step S144 is continued to be executed in a loop.

[0143] Step S146: If the physical consistency loss is less than the preset threshold, the loop is exited and the candidate enhanced image and the candidate multi-modal product generated this time are taken as the enhanced image and the multi-modal product that pass the verification.

[0144] In an optional embodiment of the present application, as shown in Figure 1c After step S14, the method of the present application further comprises:

[0145] Step S15: According to the enhanced causal parameter vector corresponding to the enhanced image and the multi-modal product that pass the verification, a two-dimensional label of the enhanced image is generated by using a perspective projection algorithm and a differentiable rasterization algorithm.

[0146] Optionally, step S15 specifically comprises:

[0147] Step S151: According to the scene geometry parameters in the enhanced causal parameter vector, the three-dimensional bounding box vertex coordinates of each object in the scene are obtained.

[0148] Step S152: According to the intrinsic parameters and extrinsic parameters in the camera parameters in the enhanced causal parameter vector, a perspective projection matrix is constructed.

[0149] Step S153: The three-dimensional bounding box vertex of each object is transformed to a two-dimensional image space by the perspective projection matrix.

[0150] Step S154: The minimum circumscribed rectangle of each object after projection is calculated in the two-dimensional image space to obtain the corresponding two-dimensional bounding box label.

[0151] Step S155: According to the scene geometry information in the enhanced causal parameter vector, the segmentation mask of each object is generated by differentiable rasterization;

[0152] Step S156: Obtain the pre-defined three-dimensional key point coordinates of each object, and transform the three-dimensional key point coordinates to the two-dimensional image space through the perspective projection matrix to obtain the corresponding two-dimensional key point coordinates;

[0153] Step S157: Combine the two-dimensional bounding box, segmentation mask and two-dimensional key point coordinates to form the two-dimensional label of the enhanced image.

[0154] Specifically, after obtaining the enhanced causal parameter vector , first, the three-dimensional bounding box vertex coordinates of each object j are extracted from the scene geometry parameters , and the homogeneous coordinates of the eight vertices are denoted as [X k ⁽ʲ⁾;1] where k=1,2,...,8. Then, the perspective projection matrix P=K[R|t] is constructed using the camera parameters ={K,R,t}, and each three-dimensional vertex homogeneous coordinate is transformed to a two-dimensional homogeneous coordinate by matrix multiplication ũ k =P[X k ⁽ʲ⁾;1]. The accurate pixel coordinates of the object on the image plane are obtained by perspective division u k =(ũ x / ũ z ,ũ y / ũ z ). By traversing all eight projection points, the minimum and maximum values of the x and y coordinates are taken, and the two-dimensional bounding box label of the object is automatically calculated: BBox⁽ʲ⁾=[min k u x ,min k u y ,max k u x ,max k u y ].

[0155] For the pixel-level segmentation mask label, the differentiable rasterization technique is directly used, and the three-dimensional mesh (V, F) of the object j is input together with its object identifier and camera parameters . The rasterizer outputs a binary mask M⁽ʲ⁾, in which the pixel area visible to the object j is accurately marked as 1. For the key point label, if the object j has a pre-defined three-dimensional key point set ⁽ʲ⁾, the same perspective projection transformation =(p̃ x / p̃z p̃ y p̃ z wherein, P[ ⁽ʲ⁾;1] are obtained. Finally, the automatically generated two-dimensional bounding box, segmentation mask and two-dimensional key point coordinates are combined to form a high-precision two-dimensional label that is completely registered with the augmented image. This process is completely based on physically rendered geometry, eliminating annotation errors caused by traditional image transformations.

[0156] Step S16: generating an augmented sample according to the augmented image that passes the verification, the multi-modal product and the two-dimensional label, wherein the augmented sample includes the augmented image, the two-dimensional label, the multi-modal product and related metadata.

[0157] wherein the metadata includes an augmented causal parameter vector , a perturbation , etc., and the content of the metadata can be selected by a person skilled in the art according to actual needs, which is not limited herein.

[0158] In an optional embodiment of the present application, when a video sequence is processed, the method of the present application further includes:

[0159] Step S17: imposing a smoothing prior constraint on the perturbation of the augmented causal parameter vector between adjacent frames, wherein the perturbation t of the t-th frame is generated based on the perturbation (t-1) of the (t-1)-th frame and satisfies and ;

[0160] wherein, is the perturbation of the t-th frame; is the perturbation of the (t-1)-th frame; is the perturbation change amount; is the covariance between the components in the perturbation ; is a smoothing threshold, which can be flexibly set by a person skilled in the art according to actual needs, which is not limited herein; denotes the perturbation of the t-th frame, which is randomly sampled from a Gaussian distribution (i.e., a normal distribution) with a mean of and a covariance of .

[0161] It should be noted that the steps S1101 to S1105, the steps S110 to S119, the steps S141 to S146, the steps S151 to S157 and the step S17 are only serial numbers written for the convenience of description, and therefore are not illustrated.

[0162] The image enhancement method based on causal reverse reconstruction and physical consistency constraint provided by the application firstly generates an enhanced sample with higher realism and physical rationality in terms of illumination, shadow, perspective, etc. by performing enhancement in a physical parameter space instead of traditional pixel transformation, effectively avoiding geometric distortion and optical abnormalities. Secondly, based on the automatic label reconstruction mechanism of the rendering geometry, the boundary box, segmentation mask and key point label accurately matched with the enhanced image can be generated synchronously, completely eliminating the label error accumulation problem caused by approximate transformation in the traditional enhancement method, and realizing the automatic process with zero label error. In addition, the method naturally supports consistency enhancement of multi-modal data and video sequences, and guarantees the inter-frame continuity through the temporal smoothing constraint, thereby providing high-quality training data for video understanding tasks.

[0163] Based on the same inventive concept, the embodiments of the application also provide a causal reverse reconstruction and physical consistency constraint based image enhancement device for implementing the causal reverse reconstruction and physical consistency constraint based image enhancement method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more causal reverse reconstruction and physical consistency constraint based image enhancement device embodiments provided below can be referred to the limitations of the causal reverse reconstruction and physical consistency constraint based image enhancement method described above, which will not be repeated here.

[0164] As shown in Figure 2 The application provides a causal reverse reconstruction and physical consistency constraint based image enhancement device, which comprises a parameter reconstruction module 21, a parameter enhancement module 22, a rendering module 23 and a verification module 24.

[0165] The parameter reconstruction module 21 is configured to reversely reconstruct a causal parameter vector from an input image, and calculate an optimal causal parameter vector according to the causal parameter vector and a target function. The causal parameter vector comprises scene geometry, material attribute, illumination parameter and camera parameter. The target function comprises a pixel matching term, a gradient domain consistency term, a physical constraint term and a regularization term.

[0166] The parameter enhancement module 22 is connected with the parameter reconstruction module, and is configured to apply a parameter perturbation term to the optimal causal parameter vector to generate an enhanced causal parameter vector. The amplitude of the parameter perturbation term is adaptively set according to the category scarcity degree.

[0167] The rendering module 23 is connected with the parameter enhancement module, and is configured to generate a candidate enhanced image and a candidate multi-modal product by using a differentiable renderer according to the enhanced causal parameter vector.

[0168] The checking module 24 is connected with the rendering module, and is configured to perform physical consistency checking on the candidate enhanced image and the candidate multi-modal product, and output the enhanced image and the multi-modal product passing the checking based on the checking result.

[0169] Optionally, the checking module 24 is specifically configured to: perform physical consistency checking on the candidate enhanced image and the candidate multi-modal product; if the physical consistency loss is less than a preset threshold, take the candidate enhanced image and the candidate multi-modal product as the enhanced image and the multi-modal product passing the checking; if the physical consistency loss is greater than or equal to the preset threshold, modify the enhanced causal parameter vector corresponding to the candidate enhanced image and the candidate multi-modal product, and cyclically execute the following rendering and checking steps until a preset output condition is met: generate the candidate enhanced image and the candidate multi-modal product by using the differentiable renderer according to the modified enhanced causal parameter vector, calculate the physical consistency loss according to the candidate enhanced image and the candidate multi-modal product, and perform physical consistency checking on the generated candidate enhanced image and the candidate multi-modal product; if the physical consistency loss is greater than or equal to the preset threshold, continue to modify the enhanced causal parameter vector corresponding to the candidate enhanced image and the candidate multi-modal product; if the physical consistency loss is less than the preset threshold, exit the loop and take the generated candidate enhanced image and the candidate multi-modal product as the enhanced image and the multi-modal product passing the checking.

[0170] Optionally, the parameter reconstruction module 21 is specifically configured to:

[0171] obtain an initial causal parameter vector from the input image;

[0172] cyclically execute the following steps:

[0173] input the current causal parameter vector into the differentiable renderer to obtain a corresponding rendered image;

[0174] calculate a pixel-level difference between the input image and the rendered image, and calculate a pixel matching term based on the pixel-level difference between the input image and the rendered image;

[0175] calculate a gradient map difference between the input image and the rendered image, and calculate a gradient domain consistency term based on the gradient map difference between the input image and the rendered image and a preset gradient domain weight;

[0176] calculate a physical consistency penalty of the current causal parameter vector based on a physical consistency constraint, and calculate a physical constraint term based on the physical consistency penalty and a preset physical constraint weight;

[0177] calculate a difference between the current causal parameter vector and a prior causal parameter vector, and calculate a regularization term based on the difference between the current causal parameter vector and the prior causal parameter vector and a preset regularization weight;

[0178] The pixel matching term, the gradient domain consistency term, the physical constraint term and the regularization term are added to obtain a target function;

[0179] It is judged whether the sequence composed of all the target functions meets a preset convergence condition;

[0180] If the sequence composed of all the target functions does not meet the preset convergence condition, the current causal parameter vector is updated based on a gradient of the target function on the current causal parameter vector, and the loop is continued;

[0181] If the sequence composed of all the target functions meets the preset convergence condition, the loop is exited, and a causal parameter vector that makes a target function in the sequence composed of all the target functions minimum is taken as an optimal causal parameter vector.

[0182] Optionally, the optimal causal parameter vector is calculated according to the following formula:

[0183]

[0184] wherein, is the optimal causal parameter vector; is a pixel matching term, representing an L1 norm of a pixel-level difference between an input image and a rendered image ; is a gradient domain weight; is an L1 norm of a gradient map difference between the input image and the rendered image ; is a gradient domain consistency term; is a physical constraint weight; is a physical consistency penalty; is a physical constraint term; is a regularization weight; is an L2 norm of a difference between a current causal parameter vector and a prior causal parameter vector ; is a regularization term; denotes the target function; denotes a causal parameter vector that makes a target function in the sequence composed of all the target functions minimum.

[0185] Optionally, the parameter enhancement module 22 is specifically configured to:

[0186] count sample quantities of each category in the training data set, and calculate a sample proportion of each category;

[0187] calculate a scarcity weight of each category according to the sample proportion, wherein the scarcity weight is negatively correlated with the sample proportion;

[0188] determine a perturbation amplitude range of each category according to the scarcity weight;

[0189] applying random perturbations to scene geometry parameters, material attribute parameters, lighting parameters and camera parameters in the optimal causal parameter vector, and taking the optimal causal parameter vector after the perturbations as an enhanced causal parameter vector.

[0190] Optionally, according to the enhanced causal parameter vector corresponding to the enhanced image and the multi-modal product passing the verification, a two-dimensional label of the enhanced image is generated through perspective projection and differentiable rasterization, including:

[0191] According to the scene geometry parameters in the enhanced causal parameter vector, obtaining the three-dimensional bounding box vertex coordinates of each object in the scene;

[0192] According to the intrinsic and extrinsic parameters in the camera parameters in the enhanced causal parameter vector, constructing a perspective projection matrix;

[0193] Transforming the three-dimensional bounding box vertex of each object through the perspective projection matrix to the two-dimensional image space;

[0194] Calculating the minimum circumscribed rectangle of each object after projection in the two-dimensional image space to obtain the corresponding two-dimensional bounding box label;

[0195] According to the scene geometry information in the enhanced causal parameter vector, generating a segmentation mask of each object through differentiable rasterization;

[0196] Obtaining the pre-defined three-dimensional key point coordinates of each object, and transforming the three-dimensional key point coordinates through the perspective projection matrix to the two-dimensional image space to obtain the corresponding two-dimensional key point coordinates;

[0197] Combining the two-dimensional bounding box, the segmentation mask and the two-dimensional key point coordinates to form the two-dimensional label of the enhanced image.

[0198] Optionally, the candidate enhanced image and the candidate multi-modal product are subjected to physical consistency verification, including:

[0199] Verifying whether the material reflection characteristic in the material attribute parameter in the enhanced causal parameter vector conforms to the law of conservation of energy;

[0200] If the material reflection characteristic does not conform to the law of conservation of energy, determining that the physical consistency loss is greater than or equal to a preset threshold, and ending the current verification;

[0201] If the material reflection characteristic conforms to the law of conservation of energy, the following verification steps are continued to be executed:

[0202] Obtaining shadow consistency loss by comparing the edge alignment degree and the intensity conservation between the candidate enhanced image and the shaded image rendered by the enhanced causal parameter vector;

[0203] By analyzing the depth map and the occlusion map in the candidate multi-modal product, whether the foreground object depth is less than the background object depth is verified, and an occlusion order loss is obtained;

[0204] The shadow consistency loss and the occlusion order loss are combined according to weights, and a physical consistency loss is obtained.

[0205] Optionally, when processing a video sequence, the device further comprises a smoothing verification module (not shown in the figure) connected with the verification module, configured to impose a smoothing prior constraint on the enhanced causal parameter vector perturbation between adjacent frames, wherein the perturbation Δθ t Based on the perturbation Δθ (t-1) is generated and satisfies and ; wherein, is the perturbation of the t-th frame; is the perturbation of the t-1-th frame; is the perturbation change amount; is the covariance between each component in the perturbation ; and is a smoothing threshold, which can be flexibly set by those skilled in the art according to actual needs, and is not limited here; indicates that the perturbation of the t-th frame is randomly sampled from a Gaussian distribution (i.e. normal distribution) with a mean of and a covariance of .

[0206] The image enhancement device based on causal reverse reconstruction and physical consistency constraint of the application firstly generates enhanced samples with higher realism and physical rationality in terms of illumination, shadow, perspective, etc. by enhancing in the physical parameter space instead of traditional pixel transformation, effectively avoiding geometric distortion and optical anomaly. Secondly, based on the automatic label reconstruction mechanism of the rendering geometry, the boundary box, the segmentation mask and the key point label accurately registered with the enhanced image can be generated synchronously, completely eliminating the label error accumulation problem caused by approximate transformation in traditional enhancement methods, and realizing the zero-label-error automatic process. In addition, this method naturally supports consistency enhancement of multi-modal data and video sequences, guarantees the inter-frame continuity through the temporal smoothing constraint, and provides high-quality training data for video understanding tasks.

[0207] It should be noted that in the application, a plurality of includes two or more.

[0208] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0209] Each module in each device in the present application can be realized by software, hardware and a combination thereof in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0210] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 3 The computer device includes a processor, a memory and a network interface connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store data required or generated for executing the above-mentioned image enhancement method based on causal reverse reconstruction and physical consistency constraint. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an image enhancement method based on causal reverse reconstruction and physical consistency constraint.

[0211] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram thereof can be as shown in Figure 3As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless mode can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement an image enhancement method based on causal reverse reconstruction and physical consistency constraint. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad provided on the shell of the computer device. It can also be an external keyboard, touchpad or mouse, etc.

[0212] Those skilled in the art can understand that, Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0213] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0214] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0215] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0216] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0217] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0218] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0219] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. An image enhancement method based on causal inverse reconstruction and physical consistency constraints, characterized in that, include: The causal parameter vector is reconstructed from the input image in reverse, and the optimal causal parameter vector is calculated based on the causal parameter vector and the objective function. The causal parameter vector includes scene geometry parameters, material property parameters, lighting parameters and camera parameters, and the objective function includes pixel matching term, gradient domain consistency term, physical constraint term and regularization term. A parameter perturbation term is applied to the optimal causal parameter vector to generate an enhanced causal parameter vector, wherein the magnitude of the parameter perturbation term is adaptively set according to the class scarcity. Based on the enhanced causal parameter vector, a differentiable renderer is used to generate candidate enhanced images and candidate multimodal products; Physical consistency verification is performed on candidate enhanced images and candidate multimodal products, and based on the verification results, the enhanced images and multimodal products that pass the verification are output. The step of reconstructing the causal parameter vector from the input image inversely, and calculating the optimal causal parameter vector based on the causal parameter vector and the objective function, includes: Obtain the initial causal parameter vector from the input image; Repeat the following steps: Input the current causal parameter vector into the differentiable renderer to obtain the corresponding rendered image; Calculate the pixel-level differences between the input image and the rendered image, and calculate pixel matching terms based on the pixel-level differences between the input image and the rendered image; Calculate the gradient map difference between the input image and the rendered image, and calculate the gradient domain consistency term based on the gradient map difference between the input image and the rendered image and the preset gradient domain weights; Based on physical consistency constraints, calculate the physical consistency penalty of the current causal parameter vector, and calculate the physical constraint terms based on the physical consistency penalty and the preset physical constraint weights. Calculate the difference between the current causal parameter vector and the prior causal parameter vector, and calculate the regularization term based on the difference between the current causal parameter vector and the prior causal parameter vector and the preset regularization weight. The objective function is obtained by adding the pixel matching term, gradient domain consistency term, physical constraint term, and regularization term. Determine whether the sequence of all objective functions meets the preset convergence conditions; If the sequence of all objective functions does not meet the preset convergence condition, update the current causal parameter vector based on the gradient of the objective function with respect to the current causal parameter vector, and continue the loop. If the sequence of all objective functions meets the preset convergence condition, exit the loop and take the causal parameter vector that minimizes the objective function in the sequence of all objective functions as the optimal causal parameter vector. The step of applying a parameter perturbation term to the optimal causal parameter vector to generate an enhanced causal parameter vector includes: Count the number of samples in each category in the training dataset and calculate the sample proportion of each category; Based on the sample proportion, the scarcity weight of each category is calculated, where the scarcity weight is negatively correlated with the sample proportion; Based on the scarcity weight, determine the perturbation range for each category; Random perturbations are applied to the scene geometry parameters, material property parameters, lighting parameters, and camera parameters in the optimal causal parameter vector, and the optimal causal parameter vector after perturbation is used as the enhanced causal parameter vector.

2. The method according to claim 1, characterized in that, After performing physical consistency verification on the candidate enhanced image and candidate multimodal product, and outputting the enhanced image and multimodal product that pass the verification based on the verification result, the method further includes: Two-dimensional labels for the enhanced images are generated by perspective projection and differentiable rasterization based on the enhanced causal parameter vectors corresponding to the verified enhanced images and multimodal products. Enhanced samples are generated based on the validated enhanced images, multimodal products, and 2D labels. The enhanced samples include the enhanced images, 2D labels, multimodal products, and related metadata.

3. The method according to claim 1 or 2, characterized in that, The step of performing physical consistency verification on candidate enhanced images and candidate multimodal products, and outputting enhanced images and multimodal products that pass the verification based on the verification results, includes: Physical consistency verification is performed on candidate enhanced images and candidate multimodal products; If the physical consistency loss is less than a preset threshold, the candidate enhanced image and candidate multimodal product are taken as the enhanced image and multimodal product that pass the verification. If the physical consistency loss is greater than or equal to a preset threshold, correct the enhancement causal parameter vectors corresponding to the candidate enhanced image and candidate multimodal product, and repeatedly execute the following rendering and verification steps until the output conditions are met: Based on the corrected enhanced causal parameter vector, a differentiable renderer is used to generate candidate enhanced images and candidate multimodal products. Based on the candidate enhanced images and candidate multimodal products, the physical consistency loss is calculated, and the physical consistency of the generated candidate enhanced images and candidate multimodal products is verified. If the physical consistency loss is greater than or equal to a preset threshold, the enhancement causal parameter vectors corresponding to the candidate enhanced image and the candidate multimodal product are corrected, and the loop continues. If the physical consistency loss is less than a preset threshold, the loop is exited and the candidate enhanced image and candidate multimodal product generated this time are taken as the enhanced image and multimodal product that pass the verification.

4. The method according to claim 1, characterized in that, The optimal causal parameter vector is calculated using the following formula: ;in, This represents the optimal causal parameter vector. For pixel matching terms, representing the input image With rendering images The L1 norm of pixel-level differences; Weights in the gradient domain; For the input image With rendering images The L1 norm of the gradient plot difference; For gradient domain consistent terms; For physical constraint weights; Penalty for physical consistency; For physical constraints; For regularization weights; For the current causal parameter vector With the prior causal parameter vector The L2 norm of the difference; For regularization terms; Represent the objective function; This represents the causal parameter vector that minimizes the objective function in the sequence of all objective functions.

5. The method according to claim 2, characterized in that, The step of generating two-dimensional labels for the enhanced image based on the enhanced causal parameter vector corresponding to the verified enhanced image and multimodal product, through perspective projection and differentiable rasterization, includes: Based on the scene geometry parameters in the enhanced causal parameter vector, obtain the 3D bounding box vertex coordinates of each object in the scene; Construct a perspective projection matrix based on the intrinsic and extrinsic parameters of the camera parameters in the enhanced causal parameter vector; Transform the vertices of the 3D bounding box of each object to a 2D image space using a perspective projection matrix; Calculate the minimum bounding rectangle of each object after projection in the two-dimensional image space to obtain the corresponding two-dimensional bounding box label; Based on the scene geometry information in the enhanced causal parameter vector, a segmentation mask for each object is generated by differentiable rasterization. Obtain the predefined 3D key point coordinates for each object, and transform the 3D key point coordinates to 2D image space through a perspective projection matrix to obtain the corresponding 2D key point coordinates; The two-dimensional bounding box, segmentation mask, and two-dimensional key point coordinates are combined to form a two-dimensional label for the enhanced image.

6. The method according to claim 3, characterized in that, The physical consistency verification of candidate enhanced images and candidate multimodal products includes: Verify whether the material reflection properties in the material attribute parameters of the enhanced causal parameter vector conform to the law of conservation of energy; If the material's reflective properties do not conform to the law of conservation of energy, the physical consistency loss is determined to be greater than or equal to a preset threshold, and the current verification ends. If the material's reflective properties conform to the law of conservation of energy, continue with the following verification steps: The shadow consistency loss is obtained by comparing the edge alignment and intensity conservation between the candidate enhanced image and the shading map rendered by the enhanced causal parameter vector; By analyzing the depth map and occlusion map in the candidate multimodal products, we can verify whether the depth of the foreground object is less than the depth of the background object and obtain the occlusion orderliness loss. The physical consistency loss is obtained by combining the shadow consistency loss and the occlusion ordering loss according to their weights.

7. The method according to claim 1, characterized in that, When processing video sequences, the method further includes: Apply a smoothing prior constraint to the enhanced causal parameter vector perturbation between adjacent frames, where, starting from the second frame, the perturbation Δθ in frame t is... t Based on the perturbation Δθ of the (t-1)th frame (t-1) Generate, and satisfy and ; in, Let be the perturbation in frame t; For the perturbation of the (t-1)th frame; The change is the disturbance. For disturbance The covariance between the components in the equation; The smoothing threshold; Indicates the perturbation in frame t. From a mean Covariance is It is obtained by random sampling from a Gaussian distribution.

8. An image enhancement device based on causal inverse reconstruction and physical consistency constraints, characterized in that, include: The parameter reconstruction module is used to inversely reconstruct a causal parameter vector from the input image and calculate the optimal causal parameter vector based on the causal parameter vector and the objective function. The causal parameter vector includes scene geometry, material properties, lighting parameters, and camera parameters. The objective function includes pixel matching terms, gradient domain consistency terms, physical constraint terms, and regularization terms. The inverse reconstruction of the causal parameter vector from the input image and the calculation of the optimal causal parameter vector based on the causal parameter vector and the objective function includes: obtaining an initial causal parameter vector from the input image; cyclically executing the following steps: inputting the current causal parameter vector into a differentiable renderer to obtain the corresponding rendered image; calculating the pixel-level difference between the input image and the rendered image, and calculating the pixel matching term based on the pixel-level difference between the input image and the rendered image; calculating the gradient map difference between the input image and the rendered image, and calculating the gradient domain based on the gradient map difference between the input image and the rendered image and a preset gradient domain weight. Consistency term; Based on physical consistency constraints, calculate the physical consistency penalty of the current causal parameter vector, and calculate the physical constraint term based on the physical consistency penalty and the preset physical constraint weights; Calculate the difference between the current causal parameter vector and the prior causal parameter vector, and calculate the regularization term based on the difference between the current causal parameter vector and the prior causal parameter vector and the preset regularization weights; Add the pixel matching term, gradient domain consistency term, physical constraint term, and regularization term to obtain the objective function; Determine whether the sequence of all objective functions meets the preset convergence condition; If the sequence of all objective functions does not meet the preset convergence condition, update the current causal parameter vector based on the gradient of the objective function with respect to the current causal parameter vector, and continue the loop; If the sequence of all objective functions meets the preset convergence condition, exit the loop, and take the causal parameter vector that minimizes the objective function in the sequence of all objective functions as the optimal causal parameter vector; The parameter enhancement module, connected to the parameter reconstruction module, is used to apply parameter perturbation terms to the optimal causal parameter vector to generate an enhanced causal parameter vector. The magnitude of the parameter perturbation term is adaptively set based on the class scarcity. The process of applying parameter perturbation terms to the optimal causal parameter vector to generate the enhanced causal parameter vector includes: counting the number of samples in each class in the training dataset and calculating the sample proportion of each class; calculating the scarcity weight of each class based on the sample proportion, where the scarcity weight is negatively correlated with the sample proportion; determining the perturbation magnitude range for each class based on the scarcity weight; applying random perturbations to the scene geometry parameters, material property parameters, lighting parameters, and camera parameters in the optimal causal parameter vector, and using the perturbated optimal causal parameter vector as the enhanced causal parameter vector. The rendering module, connected to the parameter enhancement module, is used to generate candidate enhanced images and candidate multimodal products using a differentiable renderer based on the enhancement causal parameter vector. The verification module, connected to the rendering module, performs physical consistency verification on candidate enhanced images and candidate multimodal products, and outputs the enhanced images and multimodal products that pass the verification based on the verification results.

Images