Image inpainting method, system, medium and device based on reverse cascade energy structure cooperative evolution

Abstract

The application provides an image restoration method and system based on reverse cascade energy structure cooperative evolution, a medium and equipment, belonging to the technical field of image processing and image restoration. The system comprises: a front-end CNN for visual feature extraction to obtain an initial feature map; a distance map generator for calculating the shortest Euclidean distance from each pixel in the damaged area to the known area to obtain a distance map; and dividing the damaged area into equidistant concentric ring-shaped bands; a boundary statistical encoder for calculating the statistical moments of each ring-shaped band and the joint features of each pixel in each ring-shaped band; a reverse band cascade HDNN for calculating the Hamiltonian of each ring-shaped band using a Hamiltonian function, performing half-step symplectic evolution in order from the outer band to the inner band; a Transform repair network for using the evolution results as input, using the Transform repair network to repair the damaged area, and outputting a repaired image. The scheme can guarantee the structural consistency of the damaged image restoration, avoid distortion, and improve the repair effect.

Description

Technical Field

[0001] This application relates to the field of image processing and image restoration technology, and in particular to an image restoration method, system, medium and device based on the cooperative evolution of reverse cascaded energy structures. Background Technology

[0002] Existing image inpainting methods are usually based on deep convolutional neural networks (CNN-Transformer) or generative adversarial networks (GAN-Inpainting) frameworks, which model the image inpainting task as a one-way mapping process from a damaged image to a complete image. In such methods, the model learns contextual features to directly infer the pixel content of the missing region, thereby achieving semantic continuity and texture completion of the image.

[0003] However, when the area to be repaired is large and its internal information is severely lacking, traditional models rely on overall feature extraction and feedback mechanisms to map the information of the complete image region to the damaged area in a holistic and one-time manner to obtain the repaired image. This leads to deviations in the repair results in terms of detail restoration and structure preservation. The possible reasons are as follows: On the one hand, the feature extraction of the complete image region information is affected by global normalization or convolutional smoothing, and the original brightness and color distribution are averaged. Mapping it as a whole to the damaged area results in problems such as reduced brightness and color drift at the edges. On the other hand, the central region of the damaged image lacks effective contextual constraints. In order to maintain overall consistency, the model continuously enhances the generation intensity through feedback, thereby forming brightness accumulation or overexposure artifacts. Summary of the Invention

[0004] The purpose of this application is to provide an image restoration method, system, medium, and device based on the cooperative evolution of reverse cascaded energy structures, so as to solve or alleviate the problems of detail restoration and structural errors in the image restoration of images with large missing areas and severe loss of internal information in the existing technology.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] In a first aspect, this application provides an image inpainting system based on the cooperative evolution of reverse cascaded energy structures, comprising:

[0007] The front-end CNN is used to extract visual features from pre-made sample data to obtain initial feature maps. The sample data includes images. ;image It contains both damaged and known areas;

[0008] A distance map generator is used to calculate the shortest Euclidean distance from each pixel in the damaged area to the known area to obtain a distance map; and based on the distance map, the damaged area is divided into equidistant concentric ring bands, and a band index map is output.

[0009] A boundary statistical encoder is used to input data based on the indexed map and the initial feature map. and images The statistical moments of each annular band and the joint features of each pixel in each annular band are calculated.

[0010] The reverse-band cascaded HDNN is used to calculate the Hamiltonian of each annular band based on the statistical moments and the joint features using the Hamiltonian function. The statistical moments of the outer band are used as the boundary condition energy terms of the inner band. Half-step symplectic evolution is performed in the order from the outer band to the inner band to obtain the evolution result. The outer band and the inner band are two adjacent annular bands, with the outer band closer to the edge of the damaged area and the inner band closer to the center of the damaged area.

[0011] The Transformer Repair Network is used to repair damaged areas by taking the evolution results as input and outputting a repaired image.

[0012] In the above scheme, the damaged area is divided into equidistant concentric ring bands by a distance map generator. Based on the statistical moments and joint features of each ring band, a reverse-band cascaded HDNN performs symplectic evolution based on band-level energy conservation, realizing band-by-band energy transfer from the outer band to the inner band. Utilizing the natural gradient features of the damaged area information gradually decreasing from the outer to the inner, the rich information of the outer band is propagated backward band by band to the center of the damaged image, ensuring the consistency of brightness and color between the damaged area and the known area. Simultaneously, through the synergistic constraint evolution of statistical moments and joint features, and the mathematical properties of half-step symplectic evolution, the convergence of the evolution process is guaranteed, allowing energy transfer and structural restoration to proceed synchronously. This alleviates the problem of disconnected repair logic caused by the separation of energy and structure in existing technologies, and improves the structural coherence between the damaged area and the known area. The statistical moments and joint features of each ring band calculated by the boundary statistical encoder provide effective structural and energy constraints for each ring band, enabling the repair process to fully utilize image information and avoid texture blurring or structural distortion. Using the outer band statistical moments as the inner band boundary condition energy terms, and combining the evolutionary order from the outer band to the inner band, a cascade chain of energy transfer bands is constructed, which makes energy transfer constrained by the known region, ensuring the stability and predictability of the repair process.

[0013] In conjunction with the first aspect, preferably, the reverse-band cascaded HDNN includes: an energy encoder;

[0014] The energy encoder is used to determine the initial feature map. The joint features are used to calculate the structural energy value of the current band and the energy required to generate all pixels in the current band; the calculation formulas are as follows:

[0015] , ,

[0016] In the formula, This is the current sequence number. This represents the current structural energy value. This represents the energy required to generate all pixels in the current band. This represents a 1×1 convolution kernel. For the initial feature map The current feature map is in the middle. For the parameter matrix, For the joint feature, , The current pixel in the band carried by the joint feature momentum, The current pixel in the band carried by the joint feature Normalized coordinates, The question asks for the square of the L2 norm.

[0017] In the above scheme, the energy encoder quantifies the energy at the band level by calculating the structural energy value of the current band (referred to as structural energy, corresponding to the structural state) and the energy consumption for generating the current band pixel (referred to as generation energy). This provides a clear energy input and consumption benchmark for the Hamiltonian subsystem of each band, and provides a prerequisite for energy-structure co-evolution.

[0018] In conjunction with the first aspect, preferably, the reverse-band cascaded HDNN further includes: a soft-wall unit, which is used to calculate the boundary condition energy term based on the statistical moments and joint features of the previous annular band, using the Mahalanobis distance, as shown in the following formula:

[0019] ,

[0020] In the formula, Represents the boundary condition energy term. , The statistical moments of the previous annular zone, The mean, Let covariance matrix be the variance matrix. It is a constant. It is the identity matrix. This indicates the matrix transpose.

[0021] In the above scheme, the soft wall unit is based on the statistical moments (mean) of the previous zone. Covariance The boundary condition energy term is calculated, and a computable mathematical form is designed for the outer-zone statistical moments as the inner-zone boundary condition energy term. This allows the structure-energy characteristics of the outer zone to constrain the evolution process of the inner zone, thereby enabling the joint characteristics of the inner zone to converge towards the distribution characterized by the outer-zone statistical moments during evolution. This avoids structural or texture jumps between zones and ensures the overall consistency of the restoration results. Furthermore, since the statistical moments are calculated zone by zone during the evolution from the outside to the inside, the evolution of each zone is constrained by the previous zone, preventing the disordered diffusion of energy and structure and ensuring the controllability of the restoration.

[0022] In conjunction with the first aspect, preferably, the expression for the Hamiltonian function is as follows:

[0023] ,

[0024] In the formula, For Hamiltonian, The abbreviation is , representing the boundary energy value.

[0025] In the above scheme, structural energy, generation energy, and boundary condition energy are integrated into a Hamiltonian, achieving three-dimensional synergistic constraints. This ensures that the in-band evolution process simultaneously satisfies structural consistency, energy conservation, and out-of-band constraints. Furthermore, the Hamiltonian, as the core objective function of in-band evolution, provides a clear energy-structure synergistic optimization direction for half-step symplectic evolution, ensuring that the evolution of each pixel within the band aims to minimize the Hamiltonian, thus guaranteeing the rationality of structural recovery.

[0026] In conjunction with the first aspect, preferably, the system further includes: an energy and perception evaluation module, which is used to calculate five types of losses in parallel during the training phase, sum the five types of losses in a weighted manner to obtain a joint loss, and use the joint loss for training until the model converges;

[0027] The five types of loss include: pixel loss, whole-image perceptual loss, band-level perceptual loss, out-of-band gram style loss, and energy conservation loss.

[0028] The energy and perception evaluation module is further configured to: ensure that the whole-image perception loss only updates the weights of the decoder of the Transformer repair network, block updates to the encoder weights of the Transformer repair network, and truncate the band-level perception loss so that the band-level perception loss is not passed into the inverse band-cascaded HDNN.

[0029] In the above scheme, multi-dimensional constraints of pixels, perception, band level, style, and energy are achieved through parallel computation and weighted summation of five types of losses. The whole-image perception loss only updates the Transformer decoder weights and blocks encoder updates. This not only utilizes the perception loss to optimize the visual effect of the final restoration result, but also avoids interfering with the encoder's feature extraction process of the evolution result, thus ensuring input quality. By truncating the band level perception loss and not passing it into the inverse band cascaded HDNN, the band level perception loss is prevented from destroying the energy-structure co-evolution physical logic of the HDNN, thus ensuring the functional independence and evolutionary stability of the inverse band cascaded HDNN.

[0030] In conjunction with the first aspect, preferably, the formula for calculating the energy conservation loss is as follows:

[0031] ,

[0032] In the formula, Loss due to energy conservation This represents the total number of annular bands. This is the current annular zone number. , These are the Hamiltonians before and after the current symplectic evolution, respectively.

[0033] In the above scheme, the Hamiltonian L before and after symplectic evolution is calculated. 2 Norm difference transforms the in-band energy conservation into a quantifiable loss term.

[0034] In conjunction with the first aspect, preferably, the energy and perception assessment module is further configured to: during the model inference stage, treat the five types of loss as five types of quality errors, and if any error does not meet the preset threshold condition, trigger a secondary repair.

[0035] In the above scheme, five types of losses (pixel, perception, band level, style, and energy) are transformed into quality errors in the inference stage, thereby enabling comprehensive verification of the repair results. Errors are judged by preset threshold conditions, triggering secondary repair to ensure that the repair meets the standards.

[0036] Secondly, this embodiment provides an image inpainting method based on the cooperative evolution of reverse cascaded energy structures. The method is executed by the system described in any of the above embodiments and includes the following steps:

[0037] The front-end CNN extracts visual features from pre-made sample data to obtain initial feature maps. The sample data includes images. ;image It contains both damaged and known areas;

[0038] The distance map generator calculates the shortest Euclidean distance from each pixel within the damaged area to the known area to obtain a distance map; and based on the distance map, the damaged area is divided into equidistant concentric ring bands, and a band index map is output.

[0039] The boundary statistical encoder uses the indexed map and the initial feature map as its basis. and images The statistical moments of each annular band and the joint features of each pixel in each annular band are calculated.

[0040] The reverse-band cascaded HDNN calculates the Hamiltonian of each annular band using the Hamiltonian function based on the statistical moments and joint features. It performs half-step symplectic evolution based on band-level energy conservation in the order from the outer band to the inner band to obtain the evolution result. The outer band and the inner band are two adjacent annular bands, with the outer band closer to the edge of the damaged area and the inner band closer to the center of the damaged area.

[0041] The Transformer Repair Network takes the evolution results as input, repairs the damaged areas, and outputs a repaired image.

[0042] Thirdly, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0043] Fourthly, an electronic device includes: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in any of the above embodiments.

[0044] The technical effects of the second, third, and fourth aspects of this application can be referred to the description of the first aspect, and will not be repeated here. Attached Figure Description

[0045] Figure 1 This is a logic flowchart of an image inpainting method based on the cooperative evolution of reverse cascaded energy structures, provided according to some embodiments of this application.

[0046] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0047] The embodiments of this application will now be described with reference to the accompanying drawings.

[0048] This embodiment provides an image restoration system based on the cooperative evolution of reverse cascaded energy structures. This system can be integrated into devices such as GPUs, graphics processors, image processors, vision processors, graphics card chips, and graphics card chips, providing image restoration capabilities. (Refer to...) Figure 1 The system comprises the following components: a front-end CNN, a distance graph generator, a boundary statistical encoder, a reverse-band cascaded HDNN, and a Transformer repair network.

[0049] The front-end CNN is used to extract visual features from pre-made sample data to obtain initial feature maps. Sample data includes images. ;image The dataset contains both damaged and known regions; a distance map generator calculates the shortest Euclidean distance from each pixel within the damaged region to the known region, generating a distance map; based on the distance map, the damaged region is divided into equidistant concentric ring bands, outputting a band index map; a boundary statistical encoder is used to process the band index map and the initial feature map. and images The system calculates the statistical moments of each annular band and the joint features of each pixel in each annular band; a reverse-band cascaded HDNN is used to calculate the Hamiltonian based on the statistical moments and joint features using the Hamiltonian function, and performs half-step symplectic evolution based on band-level energy conservation to obtain the evolution results; a Transformer repair network is used to repair the damaged areas using the evolution results as input, and outputs the repaired image.

[0050] In this embodiment, a front-end CNN is used to extract initial visual features, providing high-quality input for the subsequent evolution process. The step-by-step evolution of the inverse band cascaded HDNN is combined with the attention mechanism of the Transformer inpainting network, which utilizes both the physical constraints of the Hamiltonian system and the fitting ability of deep learning to complex features, to achieve high-quality restoration of missing images. By combining feature extraction, band-level partitioning, statistical encoding, and energy evolution inpainting steps, the restoration process is constrained from macroscopic structure to microscopic pixels, effectively preserving the detailed features of the long-tail distribution of the image (such as texture details, local color changes, etc.), avoiding over-smoothing, and outputting a restored image that is both realistic and complete.

[0051] In this embodiment, before performing image inpainting, sample data is first prepared, including: acquiring a complete image dataset, using a pre-made mask or a randomly generated mask on-site, and overlaying it with the complete image to obtain an image with damaged areas (referred to as a damaged image). Simultaneously, the location information (mask position) of the damaged areas (also called missing areas) and the image ground truth are recorded to obtain sample data, which includes the image with damaged areas. And truth value (label) .

[0052] The front-end CNN network is used for visual feature extraction to obtain the initial feature map. Initial feature map As the entry point embedding for the subsequent Transformer network and also as the input to the boundary statistical encoder, this feature map itself does not make category judgments or directly participate in pixel reconstruction.

[0053] The sample data is input into the front-end CNN network to extract the initial features of the whole image. Furthermore, the initial features can be the features extracted by the intermediate layers of the CNN network (referred to as intermediate visual features). Compared with the shallow features (such as color and texture) near the input layer, intermediate visual features are local combinations of shallow features. Their receptive field size is moderate, which can cover a small area in the image and describe the general visual structure. Selecting features at this level as initial features provides a better starting point for subsequent energy evolution.

[0054] It should be noted that the front-end CNN network can be any CNN network capable of feature extraction. For example, ResNet-34 is used to extract features from the complete image, and the 256-channel feature map output from ResNet-34 layer 3 (stride=16) is selected. ( ), The spatial grid size is 16×16 pixels, and the receptive field of a single cell is about 45×45 pixels. It contains edge, texture direction, color block and local shape information, without category semantics, but still retains pixel-level details.

[0055] Here, "single cell" refers to a feature map. A grid on top. A single grid receptive field refers to the feature map. The size of the area visible in the original image is represented by one grid cell. One grid cell in the feature map corresponds to a 16×16 pixel area in the original image. Theoretically, the receptive field is about 45×45 pixels. That is, the 256-dimensional vector in this grid cell is formed by compressing the edge, texture, and color block information within the approximately 45×45 range of the original image.

[0056] Original image The data is sampled into (H / 16) × (W / 16) cells by a uniform grid. For each cell, the CNN uses 256 different convolutional kernels to scan simultaneously and obtain 256 values, which are then concatenated into a 256-dimensional vector.

[0057] The distance map generator is used to calculate the shortest Euclidean distance from each pixel in the damaged area to the known area (the area without missing pixels), to obtain the distance map. Then, the damaged area is divided into multiple concentric zones according to the distance interval, providing a unique and reproducible spatial coordinate system for subsequent zone-by-zone energy conservation.

[0058] Specifically, based on the location information of the damaged area, all pixels within the damaged area are traversed, and the shortest Euclidean distance from each pixel to the nearest intact point (a point within the known area) is calculated. The calculation formula is as follows:

[0059] (1)

[0060] In the formula, For any pixel within the damaged area, For pixels To the known area The shortest distance, For known regions Any pixel within, This indicates taking the minimum value. For point With point Euclidean distance.

[0061] Shortest distance As pixels The grayscale values ​​are used to obtain a distance map, which reflects the spatial proximity of each pixel in the missing region to the known region, providing a basis for subsequent division of concentric zones according to distance intervals.

[0062] Based on the distance map, concentric bands (also known as ring bands) are generated, and a band index map is output. That is, using... As an index, the missing region is divided into equidistant concentric ring-shaped pixel sets according to a preset ring width. The rings are numbered from the outside to the inside of the damaged area, with the outermost intact pixels numbered 0, and the numbers from the outside to the inside being 0, 1, 2… -1, This represents the total number of annular zones. The boundary of each annular zone is an equidistant line, meaning that the shortest Euclidean distance between pixels on the boundary is equal.

[0063] For example, the ring width can be 1 pixel.

[0064] It should be noted that in this embodiment, the annular band can be any connected or separate region, such as a circular annular band, a rectangle, or other shapes. Since the distance is defined as the shortest distance to a known region, these bands exhibit a concentric or quasi-concentric distribution in spatial morphology, appearing as contour lines extending layer by layer into the interior of the damaged region from the boundary of the known region. These bands are referred to as concentric bands or annular bands.

[0065] Furthermore, an indexed graph can be expressed as: ,in, Indicates the first A ring-shaped zone, The range of values ​​for is [0, -1], This indicates the total number of annular bands. This means that when the ring width is 1 pixel, all conditions are met. The distance is greater than and less than or equal to All pixels with a value of +1 belong to .

[0066] By dividing the missing region into multi-level annular zones from the outside in, with the outermost zone bordering the intact pixels of the known region as a reliable energy source, and using the outer zone statistical moments as boundary condition energy terms, energy and structure are guided to be transferred unidirectionally and controllably from the outside in.

[0067] Boundary statistical encoder, used to generate an initial feature map from the output of the front-end CNN network. Image with index and original image Calculate joint features And then calculate the first Statistical moments of the belt ( , ).

[0068] Specifically, the boundary statistical encoder first applies a feature map to the CNN. Under a 16×16 grid, a ring-shaped mask is applied. Sample the boundary grid cells and obtain their center coordinates. Then return to the original image. Calculate the normalized 2D coordinates of any pixel at the corresponding position. Multidimensional (e.g., 18-dimensional) momentum This feature is then fused with the CNN feature mapped by a 1×1 convolution at that location, and concatenated to form the current joint feature. (20 dimensions), used to calculate the th Mean vector of the belt Covariance Matrix .

[0069] In this embodiment, the original image can be extracted. Multiple features of the R, G, and B channels yielded multidimensional momentum. For example, the grayscale value, x-gradient, y-gradient, Laplacian value, standard deviation of curvature, and edge direction sine and cosine of each channel are extracted, totaling 18 dimensions. The extraction method for the above features can be performed with reference to existing technologies, and will not be elaborated upon for the sake of brevity.

[0070] Then, regarding the multidimensional (18-dimensional) momentum Perform linear scaling to scale it to [-1, 1] and compare it with the normalized coordinates. The joint features that make up a single pixel, i.e. ,in, Indicates the pixel number. , These are the width and height, calculated in pixels, respectively. , Pixels before normalization coordinates Represents pixels The normalized coordinates.

[0071] Statistical processing was performed on each annular zone as a unit to calculate the mean value of momentum in each dimension within that annular zone. Covariance Thus outputting the total Set of statistical moments , This constitutes the energy-structure combined package. The calculation of the energy-structure combined package can be performed in the following sub-steps:

[0072] The set of boundary pixels for each annular band is defined as follows: , Let be the fourth neighborhood. The meaning of the above formula is: take the fourth neighborhood. Pixels in a ring band If any of its four neighboring pixels (up, down, left, right) belongs to the first... If there is a band, then determine the pixel. For boundary pixels, all boundary pixels form a set. Used for subsequent statistical analysis of each annular zone , .

[0073] Next, calculate separately , .

[0074] Specifically, based on the features of each pixel Calculate the mean vector The vector has a dimension of 20×1, and the calculation formula is as follows:

[0075] (2)

[0076] According to the characteristics of each pixel , calculate the covariance matrix , which is a positive definite matrix with a dimension of 20×20. The calculation formula is as follows:

[0077] (3)

[0078] In the formula , is the identity matrix, which is used to prevent singularity. The same applies hereinafter.

[0079] Through the above steps, a set of mean vectors and covariance matrices are calculated for each annular zone. There are a total of annular zones. Therefore, sets of statistical moments are output as the energy-structure joint package, denoted as: .

[0080] Reverse-zone cascaded HDNN (Chinese: Hamiltonian Deep Neural Network): The core idea of the reverse-zone cascaded HDNN is to deduce the generation information of the inner zone from the outer zone structure information. For the convenience of description, the serial number is used to represent the current annular zone, is used to represent the next annular zone. Since the evolution direction is from the outside to the inside, therefore, the th annular zone is also called the outer zone, and the th annular zone is also called the inner zone. The outer zone and the inner zone can be any two adjacent annular zones from the outside to the inside. By splitting the repair process from the outer zone to the inner zone into repair sub-steps according to the total number of annular zones divided, each sub-step uses the outer zone statistical moments to construct a soft wall to limit the amplitude of energy transfer, and then updates the statistical moments of the current zone through half-step symplectic evolution, and transfers them in sequence from the outside to the inside until the innermost zone is completed.

[0081] Specifically, the reverse-zone cascaded HDNN includes the following components: a structure energy encoder, a generation energy encoder, a soft wall unit, a symplectic evolution unit, and a statistical moment updater. Its input is the coordinate momentum and energy-structure joint package of the current zone.

[0082] Among them, the structure energy encoder and the generation energy encoder are collectively called the energy encoder. The structure energy encoder is used to calculate the structure energy value, and the generation energy encoder is used to calculate the generation energy value. The specific description is as follows:

[0083] The structure energy encoder is used to calculate the energy value that can be transferred by the structure information of the current zone through 1×1 convolution, that is, how much energy can be used to generate pixels, denoted as , abbreviated as: The specific calculation formula is as follows:

[0084] (4)

[0085] In the formula, This is the current annular zone; This indicates the current structural energy value of the band; This indicates a 1×1 convolution kernel, where intra-band weights are shared but inter-band weights are not. For the current CNN feature map, This represents calculating the square of the L2 norm, treating the total response as the structural energy that can be transferred in the current band.

[0086] The generation energy encoder is used to calculate the energy required to generate all pixels in the current band; that is, to calculate how much energy is needed to generate all pixels in the current band, denoted as . Abbreviation: The calculation formula is as follows:

[0087] (5)

[0088] In the formula, This indicates the consumption of energy. It is a learnable 18-dimensional diagonal matrix parameter. For the current belt pixel The momentum (i.e., the momentum of the pixels in the current band carried by the joint features) (momentum) The question asks for the square of the L2 norm.

[0089] The soft-wall element is used to calculate the boundary energy value based on the statistical moments and joint characteristics of the previous annular band, using Mahalanobis distance. As an energy valve, the soft-wall element controls the amount of energy that can pass through using statistical moments, preventing color brightness shifts and overexposure caused by a sudden influx of energy.

[0090] Specifically, the outer band (the upper ring band) , and joint features The input is fed into a soft-wall element, and a scalar is obtained by calculating the Mahalanobis distance. Since the input parameters of this scalar include: , denoted as: or The specific calculation formula is as follows:

[0091] (6)

[0092] In the formula, For joint features, including 2D normalized coordinates and 18-dimensional momentum .

[0093] It should be noted that due to pixels The set of boundary pixels from each annular band, therefore, Essentially, it refers to the joint features of the current bounded pixels, a scalar. The greater the deviation, the higher the soft wall; in other words, the scalar... The larger the value, the further the pixel deviates from the out-of-band statistical features. If the pixel is abnormal, the inverse gradient will be automatically compressed to reduce the energy entering.

[0094] The symplectic evolution unit is used to perform half-step symplectic evolution based on band-level energy conservation to obtain the evolution result, so that the energy of the outer band structure and the energy of the inner band generation can be interconverted, while the total energy remains unchanged. This unit specifically includes the following sub-steps:

[0095] First, load the tensor of the current band. , and the statistical moments of the previous annular zone ( , ), calculate the Hamiltonian of the current band using the Hamiltonian function. Abbreviation: The specific expression for the Hamiltonian function is as follows:

[0096] (7)

[0097] Record the Hamiltonian calculation results so that subsequent steps can use them for energy evolution and to calculate the energy conservation loss. Then calculate the gradient, first calculating the energy-momentum gradient:

[0098] (8)

[0099] Then calculate the energy gradient with respect to coordinates:

[0100] (9)

[0101] In the formula, 1:2 is the slicing notation, indicating that only the first two components of the joint gradient vector (corresponding to coordinates) are taken. (partial), discarded 18 dimensions (momentum) (part of the equation). Similarly, if a momentum gradient is required, take [3:20].

[0102] After calculating the gradient, a half-step update frog-leap integral loop strategy is adopted to perform half-step symplectic evolution. That is, the formulas for calculating the energy-momentum gradient and the energy-coordinate gradient are used alternately to form a half-step frog-leap integral loop. Only after completing the transformation within one band does it enter the next band for transformation again, until the transformation of all bands is completed.

[0103] Each iteration of the loop executes the following three steps sequentially:

[0104] First, let the coordinates move half a time step, and only use the gradient of the current momentum with respect to the structural energy;

[0105] Second, let the momentum travel for a full time step, and use the updated coordinates to measure the gradients of the generated energy and the soft wall energy.

[0106] Third, finally, let the coordinate move half a step further, using the already changed momentum to complete the remaining propulsion.

[0107] To simplify the formula, coordinates are... Abbreviated as , momentum Abbreviated as ,use To describe the time step of symplectic evolution, the evolutionary formulas are as follows:

[0108] (10)

[0109] (11)

[0110] (12)

[0111] In the formula, This represents the integration step size, which controls the time span of a single evolution. It is a hyperparameter with a default value of 0.1, but it is adjustable.

[0112] After the evolution of the current band is completed, its coordinates and momentum pairs [ , This process not only completes energy exchange but also maintains the total Hamiltonian within the band approximately constant, generating the coordinates and momentum of the next band, i.e. , ]. Newly generated [ , The process is repeated in the next band, and the alternation process continues until the innermost band is processed, at which point the evolution result is output. , ].

[0113] The entire alternation process continues during the training and inference phases, ensuring that energy is only controllably converted between the outer and inner bands without overall accumulation or dilution. In the above process, Hamiltonian is calculated and symplectic evolution is performed for each annular band, which is equivalent to constructing an independent Hamiltonian subsystem for each annular band, achieving synchronous conservation of structure and energy within the band during evolution.

[0114] Furthermore, in the aforementioned steps, after the current band has evolved, the statistical moment updater uses the evolved boundary pixels (pixels in the current band) to... Calculate the current band and This serves as the input to the next soft-wall element, generating a new boundary energy value.

[0115] Specifically, the statistical moment updater uses the new boundary (current band) pixel set and corresponding features, i.e., [ , As input, calculate the new (current band) mean and covariance. , The calculation results are then input into the soft-wall elements of the next zone to calculate the energy threshold of the next zone. The energy gradient of the soft-wall elements in the next zone is used to drive the energy evolution of the next zone until the innermost zone is processed, and the evolution results are output. , ].

[0116] Transformer repair network, used to evolve results [ , Using [image] as input, the damaged area is repaired to obtain the repaired image.

[0117] The Transformer inpainting network can be any image inpainting network based on Transformer technology. For example, Swin-Unet can be selected as the network skeleton of the band-level energy conservation inpainting network.

[0118] The Swin-Unet network is an implementation of Transformer. In this embodiment, the Swin-Unet network is used to complete global-local information fusion in one go, providing sufficient receptive field and detail preservation capability for subsequent band-level energy conservation repair. The Swin-Unet network reads in position tokens, content tokens, and known RGB values ​​at once, performing end-to-end regression of missing pixels while ensuring consistency in color, texture, and structure.

[0119] The details of the Swin-Unet network skeleton structure can be found in existing technologies and will not be elaborated here.

[0120] Specifically, the input data for Swin-Unet is: (as a location token) (as a content token) (Image RGB information), mask image (0 / 1 mask, 0 - missing pixel, 1 - known pixel).

[0121] It should be noted here that the image RGB information... From the original image The extracted RGB values ​​are represented by 0 for the damaged areas.

[0122] The above input data is concatenated into a 24-channel tensor (RGB occupies 3 channels). Occupying 1 lane It has 2 channels. The 18-channel image is input into the Swin-Unet network, and the output restored image is denoted as the predicted image. .

[0123] Furthermore, the inverse band cascaded HDNN also includes an energy and perception evaluation module (also known as the energy and perception evaluation network), which is used to compute five types of losses (errors) in parallel during the training phase, forming a unified gradient backpropagation path to drive the simultaneous convergence of the parameters of CNN, HDNN, and Transformer (specifically, the Swin-Unet network), ensuring that the restoration result is consistent with the ground truth in all four dimensions: numerical, perceptual, stylistic, and energy. During the inference phase, the energy and perception evaluation module calculates five types of scalar errors in real time for the restored image output from a single forward pass: pixel difference, whole-image perceptual difference, band-level perceptual difference, out-band stylistic difference, and band-level energy difference. The error value is used as a quality fingerprint to directly determine whether the current restoration result meets the preset threshold. If any indicator (i.e., the quality fingerprint) exceeds the limit, the mask is immediately re-injected and a second restoration is performed in the same process until all errors are below the threshold, ensuring that the delivered image is free of drift, overexposure, and perceptually consistent. In other words, the energy and perception evaluation module plays a dual role in multi-indicator joint quantization during training and quality gatekeeping during inference.

[0124] The five types of losses include: pixel loss, whole-image perception loss, band-level perception loss, out-of-band gram style loss, and energy conservation loss.

[0125] During the training phase, a 5-loss joint output training signal (i.e., total loss) is used. ):

[0126] (13)

[0127] in, For pixel loss, For whole-image perceptual loss, For band-level sensing loss, Due to the loss of Gram style when taking it out, Loss due to energy conservation , , , , For weights.

[0128] The following provides an example of how to calculate pixel loss, whole-image perceptual loss, band-level perceptual loss, out-of-band gram style loss, and energy conservation loss.

[0129] (1) Pixel loss .

[0130] Pixel loss By applying a Manhattan distance penalty to the RGB values ​​of each pixel, it is ensured that the repaired result does not deviate from the true color distribution, while providing non-drifting pixel anchors for subsequent energy conservation losses, preventing... It converges at a local minima where energy is conserved but color is incorrect. The formula for calculation is:

[0131] (14)

[0132] In the formula, Indicates the missing region. This represents the total number of missing pixels. Indicates missing pixels The predicted value, Indicates missing pixels The truth value of, and , ∈[0,1]³.

[0133] Pixel loss during model training The backpropagation is performed selectively, meaning that during the backpropagation process, The loss function calculates the gradient with respect to all parameters and applies a gradient truncation mechanism to prevent backpropagation into the HDNN network. Specifically, the backpropagation path is as follows: from the last 1×1 convolutional layer of Swin-Unet to the pixel space, the gradient terminates only at the Transformer output layer and does not penetrate the HDNN; at the code level, the `stop_gradient` syntax (tf.stop_gradient(...) in TensorFlow, tensor.detach() in PyTorch) is used to block the gradient from propagation into the HDNN network. , The gradient is used to avoid pixel noise disrupting energy evolution. Specifically, pixel loss... It operates on the Patch Expanding layer and skip connection bottleneck in the Swin-Unet network. By updating the upsampling kernel in the Patch Expanding layer, it aligns global texture statistics. At the same time, it performs 1×1 convolution weight fine-tuning in the skip connection bottleneck layer to maintain color consistency across multiple scales.

[0134] In this embodiment, pixel loss With energy conservation loss Synergistic effect: due to energy conservation losses It tends to generate images with reasonable structure but overall color cast, while suffering pixel loss. This strictly constrains the pixel value to be consistent with the true value. When the gradients of the two values ​​are combined and added at the parameters of the Swin-Unet decoder, it is done by giving... The loss is assigned a high weight (e.g., 1.0), giving... A lower weight (such as 0.3) is used to constrain the optimizer so that it prioritizes pixel-level color accuracy when conflicts occur.

[0135] During the inference phase, pixel loss As one of the quality fingerprints, if it exceeds a preset threshold, for example... If the value is 0.02 (normalized pixel), a secondary repair will be triggered.

[0136] (2) Whole-image perceptual loss .

[0137] Whole image perceptual loss Based on mid-level visual features, such as the feature map (F) of ResNet-34 layer 3 (stride=16, 256 dimensions), the L² squared difference between the feature map and the ground truth is calculated and the mean is taken to maintain consistency between global texture and color perception. The calculation formula is as follows:

[0138] (15)

[0139] In the formula, This refers to the repaired image, i.e., the predicted image. express The corresponding intermediate-order visual feature map, whose parameters are frozen throughout the process. Indicates the truth value. Represents the true value characteristics.

[0140] During the model training phase, whole-image perceptual loss The backpropagation path is as follows: the last layer of Swin-Unet uses a 1×1 convolution to the pixel space, updating only the decoder weights; this can be masked using the tf.stop_gradient function. The gradient of the Swin-Unet encoder is used to update only the weights of the Swin-Unet decoder to prevent perceptual loss from corrupting early edge features. Specifically applied to the PatchExpanding layer and skip connections of the Swin-Unet network, 1x1 convolutions align global texture statistics by updating the upsampling kernel, enabling seamless texture fusion between the repaired and known regions. Simultaneously, fine-tuning the 1x1 convolution weights ensures that the final output maintains color consistency with the ground truth across multiple scales, avoiding color banding or color blocks. This loss is related to... Gradient addition at the decoder layer can be achieved by... Set a higher weight, that is Greater than Ensure perception is prioritized, if Decline but An increase automatically triggers learning rate scheduling, for example... lr ×1.2, balancing the two by automatically increasing the learning rate.

[0141] In other words, whole-image perceptual loss The feature map of frozen ResNet-34 layer 3 is used as the receptive space. This feature map is frozen throughout the process and does not participate in gradient updates. An L² penalty is applied to the entire feature map to ensure that the repaired result is consistent with the ground truth in terms of texture, color, and semantic distribution, while suppressing... This could lead to a problem of energy conservation but overall oversmoothness.

[0142] During the inference phase, whole-image perception loss As a quality assessment indicator, if Greater than a preset threshold, for example If the value is > 0.05, style drift is detected, triggering a secondary repair.

[0143] (3) Band-level sensing loss .

[0144] Band-level sensing loss To extend perceptual constraints down to each annular band and achieve finer style control, the computation method is as follows: mid-level visual features (such as ResNet-34 layer 3 features) are processed according to... After slicing the mask band by band, the L² squared difference between each slice and the ground truth slice is calculated and summed to force the texture within each annular band to be consistent with the statistics of the ground truth band, thus preventing the style from migrating from the outer band to the inner band. The specific calculation formula is as follows:

[0145] (16)

[0146] In the formula, ∈{0,1}, is according to The first obtained after masking band-by-band slicing A ring-shaped binary mask, where ⊙ represents pixel-by-pixel multiplication.

[0147] Band-level perceptual loss during model training The backpropagation path is: from each scale of Swin-Unet to a band-by-band slice ( After that, the gradient only applies to the convolutional kernel corresponding to the receptive field of the band. The gradient is not passed to the HDNN, and the loss of each band only updates the part of the network in Swin-Unet responsible for reconstructing the region of that band, mainly the upsampling kernel and skip connection convolutional kernel corresponding to the receptive field, to avoid statistical conflicts between bands from disrupting energy evolution.

[0148] Band-level sensing loss Upsampling kernels and multi-scale skip connection layers are applied to the Patch Expanding layer. The upsampling kernels are fine-tuned by band weights to suppress inter-band texture leakage. Simultaneously, 1×1 bottleneck convolutional kernels with multi-scale skip connections are fine-tuned band-by-band to achieve intra-band style consistency. From the overall loss perspective... Able to This results in a double conservation, namely, Numerical energy conservation and Style statistics are conserved between bands. If the two trends are opposite (e.g., ...), Showing an upward trend If it shows a downward trend, then adjust. .

[0149] During the model inference stage, Considered as a band-level sensing error, the output is... Each band-level sensing error is used as a quality fingerprint. If the band-level sensing error of any annular band exceeds a preset threshold, for example... This will trigger a secondary repair.

[0150] (4) Out-of-band gram style loss .

[0151] Gram style loss Only the outermost annular zone (i.e., the annular zone with index 0) Calculate the difference of the gram matrices and take L². This loss can lock the edge texture statistics, providing the HDNN with a correct initial structure energy benchmark and preventing deviations caused by outermost statistical moments. The evolutionary starting point is incorrect. The specific calculation formula is as follows:

[0152] (17)

[0153] In the formula, This indicates the calculation of the Gram matrix. Indicates annular zone The mask.

[0154] During model training, The backpropagation path is: shallow layer of the Swin-Unet encoder (stride=4) → 1×1 bottleneck convolution. The gradient only updates the weights of the edge layer and does not enter the HDNN, so as to avoid the edge statistics being skewed by the energy in the back.

[0155] Specifically applied to the layer 1 convolutional kernel and skip connections of the Swin-Unet encoder: By applying the layer 1 convolutional kernel, the high-frequency response at the edges is enhanced, improving the accuracy of the initial statistical moments, which here refer to the annular bands. Statistical moments ( , At the same time, by acting on the jump connection, the outermost texture details are sharper and the initial Hamiltonian is more reliable. Here, the initial Hamiltonian refers to the annular band. The Hamiltonian obtained from the corresponding calculation is denoted as .

[0156] In model training, if Decline but An increase indicates a decrease in edge statistical error but an increase in energy loss, triggering adaptive adjustments in the soft-wall units. This automatically raises the soft-wall threshold, reducing energy spillover from the outermost band. Furthermore... and They have complementary effects: Focusing on the outermost annular zone ensures the quality of the repair starting point. It is responsible for supervising the texture statistics within each ring to ensure that the style of all areas, from the outermost to the innermost, remains consistent. The two work together to ensure style consistency from the outside to the inside through the complementary relationship of local emphasis and global coverage.

[0157] During the model inference phase, Also as one of the quality fingerprints, when Greater than the preset threshold, such as >0.01, indicating the outermost annular zone Texture distortion triggers secondary repair.

[0158] (5) Energy conservation loss .

[0159] Energy conservation loss It is used after HDNN completes the coordinate-momentum evolution of all rings, and Swin-Unet is based on the evolved token ( , The restored image obtained from regression The restored image is mapped to an 18-dimensional momentum space, and the new Hamiltonian is calculated band by band, compared with the original value. By taking the L¹ difference, the energy of the band stage is forced to be conserved, that is, the structure-energy is conserved at the band stage scale, preventing abnormal energy accumulation or collapse. The specific calculation formula is as follows:

[0160] (18)

[0161] In the formula, Indicates the first A ring-shaped band in the image restoration The Hamiltonian recalculated after re-extracting momentum. Indicates the first The original Hamiltonian before the symplectic evolution of each annular zone is calculated using the Hamiltonian function from the structural characteristics of the outer zone and the generation momentum of the inner zone. To calculate L 1 Norm, which ensures gradient continuity and robustness to outliers.

[0162] In model training, It is mainly used to optimize the parameters inside the inverse cascaded HDNN (such as the weights W of the energy encoder, the adjustable parameters ε of the soft wall unit, the symplectic evolution step size h, etc.), and its gradient is backpropagated to the input token of the HDNN. , ) place.

[0163] From the perspective of module-level functions, in the generative energy encoder, by adjusting the parameter matrix W, the contribution weight of momentum p to the generated energy is changed, thus optimizing the energy conversion efficiency between different forms; in the soft-wall unit, ε is fine-tuned to adjust the boundary conditions, avoiding the boundary conditions being too loose or rigid, which would affect the stability of energy transfer; in the symplectic evolution unit, the integral step size h is affected, optimizing the time span of the evolution process and finding the evolution rhythm that best maintains energy conservation.

[0164] It should be noted that in the Swin-Unet decoder layer, the components are... Weights of each loss , , , , Adjustments can be made as needed, and the data can be transmitted back during the training phase. For example, if... =0.3, then training is pixel / perceptual-driven with energy fine-tuning; if If the angle between the gradient and the other loss gradients is >90°, automatically perform orthogonal gradient projection, retaining only the component orthogonal to the perceptual loss to prevent parameter oscillations; when A single round of decline exceeding 15% and When the system is upgraded, a "temporarily increase soft wall threshold by 20%" strategy is triggered to prioritize the isolation of the room style.

[0165] During the model inference phase, As one of the quality fingerprints, it is used to characterize energy loss error. At this stage, Forward computation only and with Comparison, if any annular band Greater than the preset threshold, such as any band If energy is not conserved, a secondary repair is automatically triggered.

[0166] Based on the foregoing explanation, the backpropagation of the five types of loss functions can be summarized as follows: , , , The gradient is calculated at the output of the Transformer repair network, but during backpropagation, the gradient flow is truncated and does not enter the HDNN module (it is not fed into the HDNN module). , The previous propagation was only used to update the parameters of the Transformer to repair the network itself. The gradient is then fed back to the input token of the HDNN. , ) place.

[0167] In summary, the image inpainting system based on the co-evolution of inverse cascaded energy structures provided in this embodiment divides the damaged region into equidistant concentric ring bands through a distance map generator. It then combines this with an inverse-band cascaded HDNN to achieve band-by-band energy transfer from the outer band to the inner band. Utilizing the natural gradient features of the damaged region—rich in outer band information and lacking in inner band information—it solves the problems of existing inpainting systems. Figure 1 To address the issue of diluted effective energy in the inner band caused by one-time restoration, this method ensures consistency in brightness and color between the edge regions and the known regions, eliminating edge drift. A reverse-band cascaded HDNN calculates the Hamiltonian for each annular band and performs Hamiltonian evolution (symplectic evolution) to achieve synchronous conservation of energy and structure within the band. This avoids the situation of high energy accumulation in the inner band under large missing scenes, physically suppressing overexposure artifacts and ensuring the authenticity of the restored image.

[0168] Based on the same inventive concept, this application provides an image inpainting method based on the cooperative evolution of reverse cascaded energy structures. This method is executed by the image inpainting system based on the cooperative evolution of reverse cascaded energy structures provided in any of the above embodiments, and includes the following steps:

[0169] The front-end CNN extracts visual features from pre-made sample data to obtain initial feature maps. The sample data includes images. ;image It contains both damaged and known areas;

[0170] The distance map generator calculates the shortest Euclidean distance from each pixel within the damaged area to the known area to obtain a distance map; and based on the distance map, the damaged area is divided into equidistant concentric ring bands, and a band index map is output.

[0171] The boundary statistical encoder uses the indexed map and the initial feature map as its basis. and images The statistical moments of each annular band and the joint features of each pixel in each annular band are calculated.

[0172] The reverse-band cascaded HDNN calculates the Hamiltonian of each annular band based on the statistical moments and joint features using the Hamiltonian function. The statistical moments of the outer band are used as the boundary condition energy terms of the inner band. Half-step symplectic evolution is performed in the order from the outer band to the inner band to obtain the evolution result. The outer band and the inner band are two adjacent annular bands, with the outer band closer to the edge of the damaged area and the inner band closer to the center of the damaged area.

[0173] The Transformer Repair Network takes the evolution results as input, repairs the damaged areas, and outputs a repaired image.

[0174] This embodiment also provides a computer-readable storage medium (also known as a memory) storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0175] This embodiment also provides an electronic device, including: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in any of the above embodiments.

[0176] The embodiments of this application can be applied to Figure 2 The electronic device shown can be deployed at the edge as an edge device, or on the server or in the cloud as a server device. This electronic device can be a mobile terminal such as a mobile phone, tablet, handheld computer, or personal digital assistant (PDA); a smart home device such as a smart TV or smart camera; a wearable device such as a smart bracelet, smartwatch, or smart glasses; or other computer devices such as desktop, laptop, notebook, ultra-mobile personal computer (UMPC), netbook, or smart screen.

[0177] like Figure 2 As shown, the electronic device 200 may include one or more of the following components: a processor 201, a memory 203, a communication interface 202, and a communication bus 204. The memory 203 can be connected to the processor 201 via the bus 204. The bus can transfer data between the processor 201 and the memory 203. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0178] Processor 201 may include one or more processing cores. Processor 201 can connect to various parts within the electronic device 200 using various interfaces and lines. It performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 203, and by calling data stored in memory 203. For example, processor 201 may include an application processor (AP), a modem processor, a CPU, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA), and / or a neural network processing unit (NPU). The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed; the NPU implements artificial intelligence (AI) functions; and the modem handles wireless communication. Different processing units can be independent devices or integrated into one or more processors. For example, the multiple processing units shown above are all integrated into a single SoC, or the AP is a separate semiconductor chip, while other processing units are integrated into a single SoC. This application does not limit this to any particular type.

[0179] The memory 203 may include random access memory (RAM), read-only memory (ROM), or non-transitory computer-readable storage medium. The memory 203 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 203 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system or instructions for at least one function, such as an image inpainting method based on the co-evolution of reverse cascaded energy structures. The data storage area may store data created based on the use of the electronic device 200, such as sample data.

[0180] In addition, those skilled in the art will understand that the structure of the electronic device 200 shown in the above figures does not constitute a limitation on the electronic device 200. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device 200 may also include components such as a microphone, speaker, radio frequency circuit, sensor, audio circuit, power supply, and Bluetooth module, which will not be described in detail here.

[0181] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image inpainting system based on the cooperative evolution of reverse cascaded energy structures, characterized in that, include: The front-end CNN is used to extract visual features from pre-made sample data to obtain initial feature maps. The sample data includes images. ;image It contains both damaged and known areas; A distance map generator is used to calculate the shortest Euclidean distance from each pixel in the damaged area to the known area to obtain a distance map; and based on the distance map, the damaged area is divided into equidistant concentric ring bands, and a band index map is output. A boundary statistical encoder is used to input data based on the indexed map and the initial feature map. and images The statistical moments of each annular band and the joint features of each pixel in each annular band are calculated. The reverse-band cascaded HDNN is used to calculate the Hamiltonian of each annular band based on the statistical moments and the joint features using the Hamiltonian function. The statistical moments of the outer band are used as the boundary condition energy terms of the inner band. Half-step symplectic evolution is performed in the order from the outer band to the inner band to obtain the evolution result. The outer band and the inner band are two adjacent annular bands, with the outer band closer to the edge of the damaged area and the inner band closer to the center of the damaged area. The Transformer Repair Network is used to repair damaged areas by taking the evolution results as input and outputting a repaired image. The inverse band cascaded HDNN consists of the following components: a structure energy encoder, a generative energy encoder, and soft-walled units. The structural energy encoder is used to calculate the structural energy value. The specific calculation formula is as follows: ， In the formula, This is the current annular zone number; This indicates the current structural energy value of the band; This indicates a 1×1 convolution kernel, where intra-band weights are shared but inter-band weights are not. For the current CNN feature map, This represents calculating the square of the L2 norm, treating the total response as the structural energy that can be transferred in the current band. A generation energy encoder is used to calculate the energy required to generate all pixels in the current band. The calculation formula is as follows: ， In the formula, This indicates the consumption of energy. It is a learnable 18-dimensional diagonal matrix parameter. For the current belt pixel Momentum; The soft-wall element is used to calculate the boundary energy value based on the statistical moments and joint characteristics of the previous annular zone, using the Mahalanobis distance. The specific calculation formula is as follows: ， In the formula, For joint features, including 2D normalized coordinates and 18-dimensional momentum , Represents the boundary condition energy term. , The statistical moments of the previous annular zone, The mean, Let covariance matrix be the variance matrix. It is a constant. It is the identity matrix. This indicates the matrix transpose.

2. The system according to claim 1, characterized in that, The expression for the Hamiltonian function is as follows: ， In the formula, For Hamiltonian, The abbreviation is , representing the boundary condition energy term.

3. The system according to claim 1, characterized in that, The system also includes an energy and sensing assessment module. The energy and perception evaluation module is used to calculate five types of losses in parallel during the training phase, sum the five types of losses in a weighted manner to obtain a joint loss, and use the joint loss for training until the model converges. The five types of loss include: pixel loss, whole-image perceptual loss, band-level perceptual loss, out-of-band gram style loss, and energy conservation loss. The energy and perception evaluation module is further configured to: ensure that the whole-image perception loss only updates the weights of the decoder of the Transformer repair network, block updates to the encoder weights of the Transformer repair network, and truncate the band-level perception loss so that the band-level perception loss is not passed into the inverse band-cascaded HDNN.

4. The system according to claim 3, characterized in that, The formula for calculating the energy conservation loss is as follows: ， In the formula, Loss due to energy conservation This represents the total number of annular bands. , These are the Hamiltonians before and after the current symplectic evolution, respectively.

5. The system according to claim 3, characterized in that, The energy and perception assessment module is also used to: during the model inference stage, treat the five types of loss as five types of quality errors, and trigger secondary repair if any error does not meet the preset threshold condition.

6. An image inpainting method based on the cooperative evolution of reverse cascaded energy structures, the method being executed by the system described in any one of claims 1 to 5, comprising the following steps: The front-end CNN extracts visual features from pre-made sample data to obtain initial feature maps. The sample data includes images. ;image It contains both damaged and known areas; The distance map generator calculates the shortest Euclidean distance from each pixel within the damaged area to the known area to obtain a distance map; and based on the distance map, the damaged area is divided into equidistant concentric ring bands, and a band index map is output. The boundary statistical encoder uses the indexed map and the initial feature map as its basis. and images The statistical moments of each annular band and the joint features of each pixel in each annular band are calculated. The reverse-band cascaded HDNN calculates the Hamiltonian of each annular band based on the statistical moments and joint features using the Hamiltonian function. The statistical moments of the outer band are used as the boundary condition energy terms of the inner band. Half-step symplectic evolution is performed in the order from the outer band to the inner band to obtain the evolution result. The outer band and the inner band are two adjacent annular bands, with the outer band closer to the edge of the damaged area and the inner band closer to the center of the damaged area. The Transformer Repair Network takes the evolution results as input, repairs the damaged areas using the Transformer Repair Network, and outputs a repaired image. The inverse band cascaded HDNN consists of the following components: a structure energy encoder, a generative energy encoder, and soft-walled units. The structural energy encoder is used to calculate the structural energy value. The specific calculation formula is as follows: ， In the formula, This is the current annular zone number; This indicates the current structural energy value of the band; This indicates a 1×1 convolution kernel, where intra-band weights are shared but inter-band weights are not. For the current CNN feature map, This represents calculating the square of the L2 norm, treating the total response as the structural energy that can be transferred in the current band. A generation energy encoder is used to calculate the energy required to generate all pixels in the current band. The calculation formula is as follows: ， In the formula, This indicates the consumption of energy. It is a learnable 18-dimensional diagonal matrix parameter. For the current belt pixel Momentum; The soft-wall element is used to calculate the boundary energy value based on the statistical moments and joint characteristics of the previous annular zone, using the Mahalanobis distance. The specific calculation formula is as follows: ， In the formula, For joint features, including 2D normalized coordinates and 18-dimensional momentum , Represents the boundary condition energy term. , The statistical moments of the previous annular zone, The mean, Let covariance matrix be the variance matrix. It is a constant. It is the identity matrix. This indicates the matrix transpose.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in claim 6.

8. An electronic device, characterized in that, include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as claimed in claim 6.

Images