Image reconstruction apparatus, method, electronic device and readable storage medium

Abstract

The present disclosure provides an image reconstruction device, method, electronic equipment and readable storage medium, which splits an image into multiple regions, and then performs texture feature judgment for each region, so that the generative adversarial network only generates super-resolution details in the complex texture region, reduces artifacts and improves the quality of the image. The device comprises: a processing module, which extracts target region features after dividing a target image into multiple sub-images; a global discrimination module, which compares the similarity of the target region features with preset region features, determines the image type of the sub-image and sets the corresponding first loss weight; a local discrimination module, which is used for performing region feature discrimination processing on the target region features, obtaining a region probability map corresponding to the sub-image and setting the corresponding second loss weight; and a reconstruction module, which is used for generating a sub-reconstruction image corresponding to each sub-image through the first loss weight and the second loss weight, and generating a corresponding reconstruction image according to the sub-reconstruction image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and particularly relates to an image reconstruction device and method, an electronic device and a readable storage medium. BACKGROUND

[0002] Resolution is an important index for evaluating image quality, which determines the fineness of the image. In the prior art, a generative adversarial network can be used to perform super-resolution reconstruction on an image to generate a super-resolution image with higher precision. However, since the image has multiple different texture regions, the generative adversarial network cannot process different texture regions separately, resulting in artifacts in the regular texture region of the generated super-resolution image, which destroys the overall visual perception of the image. SUMMARY

[0003] The present disclosure provides an image reconstruction device and method, an electronic device and a readable storage medium. The original image is divided into multiple regions, and then feature judgment is performed on each region: if it is judged to be a complex texture region, the intensity of super-resolution processing by the generative adversarial network is increased to generate more details; if it is judged to be a regular texture region, the intensity of super-resolution processing by the generative adversarial network is reduced; the present disclosure makes the generative adversarial network generate more super-resolution details only in the complex texture region, reduces artifacts, and improves the visual perception quality of the reconstructed image.

[0004] In a first aspect, the present disclosure provides an image reconstruction device, comprising:

[0005] A processing module is configured to divide a target image into multiple sub-images and extract target region features contained in each sub-image;

[0006] A global discrimination module is configured to compare the target region features with preset regular region features in terms of similarity for each sub-image, determine an image type of the sub-image according to a comparison result, and set a first loss weight corresponding to the sub-image according to the image type; wherein the first loss weight is used to represent a global feature loss of the sub-image;

[0007] A local discrimination module is configured to perform region feature discrimination processing on the target region features for each sub-image to obtain a region probability map corresponding to the sub-image, and set a second loss weight corresponding to the sub-image according to the region probability map; wherein the second loss weight is used to represent a local feature loss of the sub-image;

[0008] A reconstruction module is configured to generate a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight corresponding to each sub-image, respectively, and generate a reconstruction image corresponding to the target image according to the sub-reconstruction images corresponding to each sub-image.

[0009] In a second aspect, the present disclosure provides an image reconstruction method, comprising:

[0010] dividing a target image into a plurality of sub-images, and extracting a target region feature contained in each sub-image;

[0011] for each sub-image, performing a similarity comparison between the target region feature and a preset rule region feature, determining an image type of the sub-image according to a comparison result, and setting a first loss weight corresponding to the sub-image according to the image type; wherein the first loss weight is used to represent a global feature loss of the sub-image;

[0012] for each sub-image, performing a region feature discrimination processing on the target region feature to obtain a region probability map corresponding to the sub-image, and setting a second loss weight corresponding to the sub-image according to the region probability map; wherein the second loss weight is used to represent a local feature loss of the sub-image;

[0013] generating a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight corresponding to each sub-image respectively, and generating a reconstruction image corresponding to the image to be processed according to the sub-reconstruction image corresponding to each sub-image.

[0014] In a third aspect, the present disclosure provides an electronic device comprising the image reconstruction apparatus described above.

[0015] In a fourth aspect, the present disclosure provides an electronic device comprising at least one processor, and a memory connected to the at least one processor in communication; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above method.

[0016] In a fifth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processing core, implements the above method.

[0017] In the embodiments provided in the present disclosure, firstly, the processing module divides the target image into a plurality of sub-images, and extracts target region features of each sub-image; then, the global discrimination module compares the target region features of each sub-image with preset rule region features in terms of similarity, and determines the image type of the sub-image according to the comparison result, and further determines the first loss weight corresponding to the sub-image; next, the local discrimination module performs region feature discrimination processing on each sub-image to obtain a region probability map corresponding to the sub-image, and further obtains the second loss weight corresponding to the sub-image; finally, the reconstruction module generates a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight of each sub-image, and generates a reconstruction image corresponding to the image to be processed according to the sub-reconstruction image corresponding to each sub-image. As can be seen, on the one hand, the present application performs global discrimination on the sub-images obtained after the target image is divided, to determine the image type of the sub-image, and generates a weight representing the global feature loss; on the other hand, the present application performs region feature discrimination on the sub-image, and calculates the weight representing the local feature loss in the form of a probability map. Compared with the image reconstruction device without target image division and double judgment, by dividing the target image, using global judgment and local judgment on each sub-image respectively and generating the corresponding weights, different texture regions can be effectively distinguished, and then different generation processing is performed according to different texture regions. In short, the way of splitting the original image into multiple regions and then performing feature judgment on each region makes the generative adversarial network generate more super-resolution details only in the complex texture region, and does not generate any details in the rule texture region (for example, the region containing object edges, rule pattern region and simple texture region, etc.), which can reduce artifacts and improve the visual perception quality of the reconstructed image.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, which together with the embodiments of the present disclosure are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art through a detailed description of the specific example embodiments, with reference to the accompanying drawings, in which:

[0020] Figure 1 a block diagram of an image reconstruction device provided for an embodiment of the present disclosure;

[0021] Figure 2 a specific structural composition schematic diagram of an image reconstruction device provided by an example of the present disclosure is shown;

[0022] Figure 3 A flowchart of an image reconstruction method provided for another embodiment of the present disclosure;

[0023] Figure 4 A block diagram of an electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] To enable persons skilled in the art to better understand the technical solutions of the present disclosure, exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of the present disclosure to help understanding, which should be considered only as exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, the description below omits the description of well-known functions and structures.

[0025] In the case of no conflict, each embodiment of the present disclosure and each feature in the embodiments can be combined with each other.

[0026] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0027] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms "comprise" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The terms "connected" or "coupled" and / or like terms are not limited to physical or mechanical connections or couplings, but can include electrical connections or couplings, whether direct or indirect.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0029] The image reconstruction method according to the embodiments of the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be a vehicle-mounted device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud computing services. The method can be implemented by calling a computer program stored in a memory by a processor.

[0030] In the related art, a generative adversarial network is usually directly used to perform super-resolution processing on an image to generate a corresponding higher-resolution image. However, since there are complex texture regions and regular texture regions in the image, the generative adversarial network will generate a large number of artifacts when processing the regular texture regions, which destroys the overall visual perception of the image. To solve the above problem, the present application proposes an image reconstruction device. First, the processing module divides the target image into multiple sub-images and extracts the target region features of each sub-image. Then, the global discrimination module compares the target region features of each sub-image with the preset regular region features in terms of similarity, and determines the image type of the sub-image according to the comparison result, and further determines the first loss weight corresponding to the sub-image. Next, the local discrimination module performs region feature discrimination processing on each sub-image to obtain the region probability map corresponding to the sub-image, and further obtains the second loss weight corresponding to the sub-image. Finally, the reconstruction module generates a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight of each sub-image, and generates a reconstruction image corresponding to the image to be processed according to the sub-reconstruction image corresponding to each sub-image. As can be seen, on the one hand, the present application performs global discrimination on the sub-images obtained by dividing the target image to determine the image type of the sub-image and generate a weight representing the global feature loss. On the other hand, the present application performs region feature discrimination on the sub-image to calculate the weight representing the local feature loss in the form of a probability map. Compared with the image reconstruction device without dividing the target image and double judgment, by dividing the target image and using global judgment and local judgment on each sub-image and generating corresponding weights, different texture regions can be effectively distinguished, and different generation processing can be performed according to different texture regions. In summary, the way of splitting the original image into multiple regions and then performing feature judgment on each region makes the generative adversarial network generate more super-resolution details only in the complex texture region, and does not generate any details in the regular texture region (such as a region containing object edges, a regular pattern region, and a simple texture region, etc.). This can reduce artifacts and improve the visual perception quality of the reconstructed image.

[0031] Figure 1 A block diagram of an image reconstruction device 10 provided for an embodiment of the present disclosure.

[0032] With reference to Figure 1 The image reconstruction device 10 comprises:

[0033] A processing module 11 is configured to divide a target image into a plurality of sub-images and extract target region features contained in each sub-image.

[0034] The target image refers to an image to be reconstructed, and the image reconstruction can obtain a super-resolution image with higher resolution and higher accuracy. Optionally, the target image is a low-resolution image. The sub-image refers to an image obtained by splitting the target image, and the target image can be obtained by splicing all the sub-images obtained by splitting. Optionally, the target image can be split into a plurality of sub-images with equal size, or the target image can be split into a plurality of sub-images with overlapping regions. The target region features refer to texture features of the sub-image, which can be divided into complex texture features and regular texture features. For example, the complex texture features exist in the random distribution of pixel points in the patterns of animal fur, leaves, grass, soil, sand, etc. The regular texture features exist in the regular grid (periodic high-frequency detail area) and fine line (simple and regular high-frequency detail area) regions, which are composed of regular structures or have steep edges between adjacent pixels. When the image region containing the regular texture features is processed using the generative adversarial network, the generated details are easily identified by the human eye as visual distortion, thereby reducing the visual perception quality. When the image region containing the complex texture features is processed using the generative adversarial network, the generated details can effectively improve the visual perception quality because the human eye is not sensitive to the difference between the random distribution of pixel points.

[0035] A global discrimination module 12 is configured to compare the target region features with preset regular region features for each sub-image, determine the image type of the sub-image according to the comparison result, and set a first loss weight corresponding to the sub-image according to the image type. The first loss weight is used to represent the global feature loss of the sub-image.

[0036] Wherein, after obtaining the target region feature contained in each sub-image, the target region feature needs to be distinguished to determine whether it belongs to a regular texture feature; similarity comparison between the target region feature and the pre-set regular region feature can determine whether the target region feature is a regular region feature, and the image type of the corresponding sub-image is obtained; wherein the image type of the sub-image includes a regular type and an irregular type; wherein the first loss weight represents the global feature loss of the sub-image, which is obtained by comparing the target region feature of the sub-image with the pre-set regular region feature.

[0037] Wherein, similarity comparison between the target region feature and the pre-set regular region feature can be realized in various ways.

[0038] For example, similarity comparison can be realized based on a direction histogram (HOG):

[0039] Firstly, the original image to which the region feature belongs is preprocessed, and the color image is converted into a gray image:

[0040] Y=rgb2gray(INPUT)

[0041] Wherein, Y represents the generated gray image, and the input of INPUT is an original image (color RGB image).

[0042] Then, gamma correction is performed to adjust the contrast of the image to reduce the influence of uneven illumination and local shadows, so that the overexposed or underexposed image is restored to normal, and is closer to the image seen by the human eye:

[0043]

[0044] Wherein, Y adj represents the gray value of the image, acts on the normalization, so that the value range of the power operation becomes [0, 1], and 255 acts on the inverse normalization, so that the value range becomes [0, 255];

[0045] Then, gradient calculation is performed on the image to calculate the amplitude and angle of the image gradient: the amplitude of the image gradient The angle of the image gradient

[0046] Wherein Wherein d x (x,y) represents the gradient value of the pixel point Y(x,y) in the horizontal direction in the gray image Y, d y (x,y) represents the gradient value of the pixel point Y(x,y) in the vertical direction in the gray image Y; Y(x,y) represents the pixel point with coordinates (x,y) in the gray image Y.

[0047] Next, the image is divided into 8x8 non-overlapping image blocks; for example, if the image size is 128x128, there are 256 image blocks; since each pixel contains the magnitude and direction of the gradient, an 8x8 image block contains 128 values; the gradient is divided into 9 values ​​at intervals of 20, where the range of the j-th direction unit is represented by [20j, 20(j+1)]; the gradient magnitude M(x,y) of each pixel is added to the j-th direction unit. The histogram corresponding to the (j+1)th directional unit is used; thus, each image patch can obtain a gradient angle histogram with 9 directional units to represent the feature attributes of the image patch, represented by HOG(m,n), where (m,n) is the position of the current image patch in the image; then, L2-norm is used to normalize the HOG:

[0048]

[0049] Here, L2-norm is a normalization calculation method used to locally normalize the contrast of all HOG(m,n) values ​​to prevent excessive variation in gradient strength; in formula f, ‖HOG(m,n)‖2 represents the L2-norm norm of HOG(m,n); and to avoid denominator... It is zero, in Add a constant e 2 .

[0050] Next, the 256x256 image is divided into 16x16 8x8 image blocks. The HOG values ​​of each image block are then concatenated from left to right and from top to bottom to obtain a 16x16x9 one-dimensional vector.

[0051] Finally, an SVM-based global texture recognizer is constructed to calculate the feature similarity between region features.

[0052] For example, a multilayer perceptron can be used as a texture recognizer, where the features are HOG and the classifier is MLP; in addition, a convolutional neural network (CNN) can be used to implement an end-to-end texture recognizer; the above two implementation methods are existing technologies and will not be elaborated here.

[0053] In one alternative implementation, similarity comparison is used to obtain the similarity of region features corresponding to the sub-image; when determining the image type of the sub-image based on the comparison result, and setting the first loss weight corresponding to the sub-image based on the image type, this is achieved in the following way:

[0054] First, the image type of the sub-image is determined based on the similarity of regional features and a preset threshold for regional feature similarity.

[0055] Then, based on the region feature similarity and the image type, the first loss weight is determined.

[0056] The image type of the sub-image includes a regular type and an irregular type, and the first loss weight of the sub-image of the regular type is greater than the first loss weight of the sub-image of the irregular type.

[0057] The region feature similarity of the target region feature is obtained by comparing the target region feature with the preset regular region feature; the preset region feature similarity threshold is a probability value; when the region feature similarity of the target region feature is greater than the preset region feature similarity threshold, it is determined that the sub-image corresponding to the target region feature is a regular type image; and when the region feature similarity of the target region feature is less than the preset region feature similarity threshold, it is determined that the sub-image corresponding to the target region feature is an irregular type image.

[0058] In a specific example, after the target region feature is discriminated, the region feature similarity between the target region feature and the preset regular region feature is obtained in the form of a probability value; for example, the preset region feature similarity threshold is 50%, and when the similarity is greater than 50%, it is determined that the sub-image corresponding to the target region feature is a regular type image, and when the similarity is less than 50%, it is determined that the sub-image corresponding to the target region feature is an irregular type image; then, according to the image type of the sub-image, the first loss weight corresponding to the sub-image is calculated in combination with the obtained region feature similarity.

[0059] In summary, the implementation mode aims to determine the image type of the sub-image through the region feature similarity and the preset threshold, and determine the first loss weight representing the global loss weight according to the region feature similarity value and the image type.

[0060] The local discrimination module 13 is configured to, for each sub-image, perform region feature discrimination processing on the target region feature to obtain a region probability map corresponding to the sub-image; and set a second loss weight corresponding to the sub-image according to the region probability map; wherein the second loss weight is used to represent the local feature loss of the sub-image.

[0061] In an optional implementation mode, when the region feature discrimination processing is performed on the target region feature for each sub-image to obtain the region probability map corresponding to the sub-image, the following method is used:

[0062] First, the sub-image is divided into a plurality of local blocks; wherein the area of any local block is smaller than the sub-image, and all the extracted local blocks can be combined into the sub-image; optionally, when the sub-image is divided to obtain the local blocks, the sub-image is divided according to the size of the pixel points, and each local block obtained is a pixel point in the sub-image.

[0063] Then, for each local block, the target local region feature corresponding to the local block is obtained; and similarity comparison is performed on the target local region feature and the preset local region feature to generate a local region feature similarity corresponding to the local block; wherein the local region feature similarity is represented by a probability value; wherein, since the sub-image contains the target region feature, after the sub-image is divided, each local block generated after the division also contains a corresponding region feature, i.e., the target local region feature; similarity comparison between the target local region feature and the preset local region feature can obtain the similarity between the target local region feature and the preset local region feature. The similarity comparison between the target local region feature and the preset local region feature can be realized in various ways. For example, the confidence that the current block is a complex and disordered texture region is calculated in a square block with the current pixel (x, y) as the center and a length of n: first, edge extraction is performed on the block to obtain img edge ; then, the absolute difference img adiff = abs(IHR-ISR) between the reconstructed image IHR and the high-definition image ISR is calculated to obtain a feature map img feat = lpf(img edge )*img adiff ; wherein the reconstructed image IHR refers to a low-resolution image that needs to be reconstructed, the high-definition image ISR refers to a high-resolution image generated after reconstruction, img adiff represents the absolute difference between the reconstructed image IHR and the high-definition image ISR; abs(IHR-ISR) represents the absolute value of the result after IHR-ISR is subtracted (to prevent negative numbers); wherein lpf(x) is a low-pass filter function, and lpf(img edge ) is used to remove noise and blur the block edge img edge ; then, the variance of each pixel point on img_feat is calculated ; wherein (x, y) is a pixel point to be calculated for variance, var(x) represents variance calculation on x; when calculating, wherein n represents the nth pixel point to be calculated for variance; the obtained variance value is used as the similarity probability value between the region feature of the current pixel point and the preset local region feature, and is used to correct the weight coefficient. For another example, a semantic segmentation neural network of a CNN network can be used to construct a local texture recognizer to perform semantic segmentation on image pixels; the pixels are divided into two categories of complex and disordered region pixels and non-complex and disordered region pixels, and the value between [0, 1] is used to represent the similarity probability value between the region feature of the current pixel point and the preset local region feature.

[0064] Finally, the local region feature similarities of all local blocks are acquired, a probability map conversion process is performed, and a region probability map corresponding to the sub-image is obtained.

[0065] In an optional implementation, when the local region feature similarities of all local blocks are acquired, the probability map conversion process is performed, and the region probability map corresponding to the sub-image is obtained, the implementation is as follows.

[0066] First, an initial probability matrix corresponding to the sub-image is constructed according to the local region feature similarities of all local blocks. After the local region feature similarities of all local blocks divided by the sub-image are acquired, the local region feature similarity values of all local blocks are filled in the matrix to form the initial probability matrix. In a specific example, the sub-image is divided into a 4x4 block grid, and a total of 16 local blocks are divided. Therefore, in the 4x4 grid, each grid represents a local block, and (x, y) is used to represent the position of each local block, that is, (1, 1) represents the local block in the first row and the first column, and (2, 3) represents the local block in the second row and the third column. After the probability values of the local region feature similarities corresponding to the local blocks are acquired, the probability values are filled in the corresponding block network to form a 4x4 matrix, which is the initial probability matrix.

[0067] Then, the initial probability matrix is smoothed to obtain a smoothed probability matrix corresponding to the initial probability matrix. Since the initial probability matrix is directly generated from the probability values of the local region feature similarities corresponding to the local blocks, calculation errors and noises may occur when the local region feature similarities are generated, resulting in abnormal fluctuations in the data. The smoothing process can eliminate data with large errors, thereby suppressing the influence of errors. Optionally, a sliding average algorithm is used to process the initial probability matrix to obtain the smoothed probability matrix.

[0068] Finally, the region probability map corresponding to the sub-image is determined according to the smoothed probability matrix; the region probability map is represented by the smoothed probability matrix. After the smoothed probability matrix is generated, the region probability map corresponding to the sub-image can be determined by the numerical distribution of the smoothed probability matrix.

[0069] The reconstruction module 14 is configured to generate a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight corresponding to each sub-image, and generate a reconstruction image corresponding to the to-be-processed image according to the sub-reconstruction image corresponding to each sub-image.

[0070] In an optional implementation, when the reconstruction image corresponding to the to-be-processed image is generated according to the sub-reconstruction image corresponding to each sub-image, the implementation is as follows.

[0071] Firstly, each sub-image and the sub-reconstruction image corresponding to each sub-image are processed to generate a first loss corresponding to a first loss weight and a second loss corresponding to a second loss weight; wherein the sub-reconstruction image is obtained by image reconstruction processing of the corresponding sub-image and has higher accuracy details than the sub-image; by processing the sub-image and the sub-reconstruction image corresponding to the sub-image, the global loss generated by global discrimination during image reconstruction and the local loss generated by local discrimination can be obtained; in a specific example, when processing the sub-image and the sub-reconstruction image, the difference method can be used to obtain the loss: the sub-image and the sub-reconstruction image are superimposed, and the difference between each pixel is obtained by difference, that is, the global loss (first loss) generated by the super-resolution processing of the sub-image by the generative adversarial network; similarly, the local blocks of the sub-image and the corresponding blocks after reconstruction are superimposed, and the difference between each pixel is obtained by difference, that is, the local loss (second loss) generated by the super-resolution processing of the sub-image by the generative adversarial network.

[0072] Then, according to the first loss weight and the first loss corresponding to the first loss weight, the second loss weight and the second loss corresponding to the second loss weight, the reconstruction loss value of each sub-reconstruction image is generated. Wherein the first loss weight is the global loss weight, and the first loss corresponding to the first loss weight is the global loss; and the second loss weight is the local loss weight, and the second loss corresponding to the second loss weight is the local loss; through the global and local two levels, the reconstruction loss value of each sub-reconstruction image in the target image is obtained respectively.

[0073] Finally, according to the reconstruction loss value of each sub-reconstruction image, the sub-reconstruction image is executed to perform correction processing; according to the corrected sub-reconstruction image, the corrected reconstruction image is generated. Wherein the reconstruction loss value is the loss value between the sub-reconstruction image generated after image reconstruction processing and the sub-image; in the multiple sub-images of the target image, when the sub-image generates super-resolution reconstruction in the complex texture area, the reconstruction loss value has positive contribution; when the sub-image generates super-resolution reconstruction in the regular texture area, the reconstruction loss value has negative contribution; because it is necessary to reduce the reconstruction intensity of the regular texture area and increase the reconstruction intensity of the complex texture area, the numerical value of the reconstruction loss value can reflect the overall image reconstruction situation. At this time, according to the reconstruction loss value of each sub-reconstruction image, each sub-reconstruction image is corrected, which can effectively reduce the artifacts generated by the reconstruction of the sub-image in the regular texture area, so as to improve the image reconstruction quality. After the correction processing of each sub-reconstruction image according to the reconstruction loss value, the corrected each sub-reconstruction image can be integrated to generate the reconstruction image corresponding to the target image.

[0074] The specific method of image reconstruction performed by the image reconstruction device provided by the present disclosure is described in detail below in a specific example manner.

[0075] It should be noted that there is a generator loss function in the adversarial generation network, which is used for secondary correction of the generated image (reconstruction loss value):

[0076] L adj = m recon · L recon + m perc · L perc + m adv · L adv + m artf · L artf

[0077] Wherein, the loss function is composed of image reconstruction loss L recon (global loss), visual perception loss L perc (local loss), adversarial loss L adv (global loss) and visual noise loss L artf (local loss), and the corresponding weight coefficients are m recon , m perc , m adv and m artf .

[0078] First, a global complex disordered texture area discriminator is constructed, the feature area contained in the target image is compared with the preset regular feature area to obtain the similarity probability value between the feature area contained in the target image and the preset regular feature area, so as to obtain the probability that the current input feature area is a complex disordered texture area, and the probability value is between [0, 1]; then the generated probability value and the image type of the target image are combined to generate the weight m adv of the adversarial loss; when the value of the weight m adv of the adversarial loss is small, it means that the current input feature area is probably not a complex and disordered texture area, so the adversarial generation network does not need to be processed at this time, and the contribution of the adversarial loss to the total loss function should be reduced. Wherein, the input feature area can be the feature area contained in the entire target image, or the feature area contained in each sub-image after the target image is divided into multiple sub-images.

[0079] Then, a local complex disordered texture region detector is constructed to extract each pixel point in the target image and compare it with a preset local regular feature region in terms of similarity to detect the probability of each pixel point in the current input feature region being in a complex disordered texture region; that is, this step is used to obtain the probability that a certain neighborhood centered on the pixel point is a complex disordered image region, which has a value range of [0, 1]; an initial probability matrix with the same size as the input image can be constructed according to the probability values corresponding to all pixel points.

[0080] Next, the initial probability matrix is smoothed to obtain a smoothed probability matrix; the smoothed probability map map_lpf can be determined according to the smoothed probability matrix.

[0081] Next, the visual perception loss weight m perc is calculated according to the preset weight λ1.

[0082] Next, the visual noise loss weight m artf is calculated according to the preset weight λ2.

[0083] Next, the corresponding visual perception loss L perc , the adversarial loss L adv , and the visual noise loss L artf are calculated.

[0084] Finally, the generator loss function L adj is constructed, and the reconstructed image generated by the target image is modified again according to the generator loss function L adj to reduce the artifacts generated by the regular texture region during reconstruction as much as possible and improve the reconstruction effect of the target image.

[0085] Further, Figure 2 a specific structural composition diagram of the image reconstruction device 10 provided by the present example is shown. As Figure 2 shown, the image reconstruction device 10 is composed of a global texture detection unit 21, a local texture detection unit 22, a local texture refinement unit 23, a artifact weight calculation unit 24, a perception loss calculation unit 25, an artifact loss calculation unit 26, and an adversarial loss calculation unit 27.

[0086] The processing flow between the modules will be described in detail below:

[0087] First, the reconstructed image (the image that needs to be reconstructed) is input into the global texture detection unit 21, and the global texture detection unit 21 detects the probability of the feature region of the current input reconstructed image being a complex disordered texture region to generate the adversarial loss weight m adv .

[0088] Then, the reconstructed image is input into the local texture detection unit 22 to detect the probability of each pixel in the current input feature region being in a complex disordered texture region;

[0089] Next, the local texture refinement unit 23 obtains the calculation parameters of the local texture detection unit 22 to generate an initial probability matrix with the same size as the reconstructed image. After smoothing the initial probability matrix, a smoothed probability matrix can be generated, and a smoothed probability map map_lpf can be obtained according to the smoothed probability matrix.

[0090] Next, the perceptual loss calculation unit 25 receives the smoothed probability map map_lpf generated by the local texture refinement unit 23, and calculates the visual perceptual loss weight m perc according to the pre-set weight λ1. At the same time, the loss between the generated image and the reconstructed image is calculated according to the input generated image to obtain the visual perceptual loss L perc .

[0091] Next, the artifact weight calculation unit 24 receives the smoothed probability map map_lpf generated by the local texture refinement unit 23, and calculates the visual noise loss weight m artf according to the pre-set weight λ2. At the same time, the artifact loss calculation unit 26 calculates the loss between the generated image and the reconstructed image according to the input generated image to obtain the visual noise loss L artf .

[0092] Next, the adversarial loss calculation unit 27 obtains the adversarial loss weight m adv generated by the global texture detection unit 21. At the same time, the loss between the generated image and the reconstructed image is calculated according to the input generated image to obtain the adversarial loss L adv .

[0093] The above generated m perc , m adv , m artf , L perc , L adv , and L artf , and the pre-generated L recon and m recon are combined to calculate the generator loss function L adj = m recon ·L recon + m perc ·L perc + m adv ·L adv + m artf ·L artf .

[0094] To sum up, in the embodiments provided in the present disclosure, first, the processing module divides the target image into a plurality of sub-images, and extracts target region features of each sub-image; then, the global discrimination module compares the target region features of each sub-image with preset rule region features in terms of similarity, and determines the image type of the sub-image according to the comparison result, and further determines the first loss weight corresponding to the sub-image; next, the local discrimination module performs region feature discrimination processing on each sub-image to obtain a region probability map corresponding to the sub-image, and further obtains the second loss weight corresponding to the sub-image; finally, the reconstruction module generates a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight of each sub-image, and generates a reconstruction image corresponding to the image to be processed according to the sub-reconstruction image corresponding to each sub-image. As can be seen, on the one hand, the present application performs global discrimination on the sub-images obtained after dividing the target image, to determine the image type of the sub-image, and generates a weight representing global feature loss; on the other hand, the present application performs region feature discrimination on the sub-image, and calculates a weight representing local feature loss in the form of a probability map. Compared with the image reconstruction device without dividing the target image and double judgment, by using global judgment and local judgment on each sub-image after dividing the target image and generating the corresponding weight, different texture regions can be effectively distinguished, and then different generation processing is performed according to different texture regions. In short, the way of splitting the original image into multiple regions and then performing feature judgment on each region makes the generative adversarial network generate more super-resolution details only in the complex texture region, reduces the artifacts, and improves the visual perception quality of the reconstruction image.

[0095] Figure 3 A flowchart of an image reconstruction method provided for another embodiment of the present disclosure.

[0096] With reference to Figure 3 The method comprises:

[0097] Step S310: dividing the target image into a plurality of sub-images, and extracting target region features contained in each sub-image.

[0098] Step S320: comparing the target region features with preset rule region features in terms of similarity for each sub-image; determining the image type of the sub-image according to the comparison result, and setting the first loss weight corresponding to the sub-image according to the image type; wherein the first loss weight is used to represent the global feature loss of the sub-image;

[0099] Step S330: performing region feature discrimination processing on the target region features for each sub-image to obtain a region probability map corresponding to the sub-image; setting the second loss weight corresponding to the sub-image according to the region probability map; wherein the second loss weight is used to represent the local feature loss of the sub-image;

[0100] Step S340: generating a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight corresponding to each sub-image; and generating a reconstruction image corresponding to the image to be processed according to the sub-reconstruction image corresponding to each sub-image.

[0101] In an optional implementation, when the region feature discrimination processing is performed on the target region feature for each sub-image to obtain a region probability map corresponding to the sub-image, the following method is used:

[0102] First, the sub-image is divided into a plurality of local blocks; the area of any local block is smaller than the sub-image, and all the extracted local blocks can be combined into the sub-image.

[0103] Then, for each local block, a target local region feature corresponding to the local block is obtained; and a local region feature similarity corresponding to the local block is generated by comparing the target local region feature with a preset local region feature; the local region feature similarity is represented by a probability value.

[0104] Finally, the local region feature similarities of all the local blocks are obtained, and a probability map conversion processing is performed to obtain the region probability map corresponding to the sub-image.

[0105] In an optional implementation, when the local region feature similarities of all the local blocks are obtained, and the probability map conversion processing is performed to obtain the region probability map corresponding to the sub-image, the following method is used:

[0106] First, for the local region feature similarities of all the local blocks, an initial probability matrix corresponding to the sub-image is constructed.

[0107] Then, the initial probability matrix is smoothed to obtain a smoothed probability matrix corresponding to the initial probability matrix.

[0108] Finally, the region probability map corresponding to the sub-image is determined according to the smoothed probability matrix; the region probability map is represented by the smoothed probability matrix.

[0109] The implementation of the image reconstruction method described above is described in the previous embodiment, and will not be repeated here.

[0110] Figure 4 A block diagram of an electronic device provided by the embodiments of the present disclosure is shown.

[0111] Referring to Figure 4The electronic device provided by the embodiment of the present disclosure includes: at least one processor 501; at least one memory 502, and one or more I / O interfaces 503 connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs executable by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501 to implement the image reconstruction method.

[0112] The embodiment of the present disclosure further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program implements the image reconstruction method when executed by a processor / processing core. The computer readable storage medium can be a volatile or non-volatile computer readable storage medium.

[0113] The embodiment of the present disclosure further provides a computer program product including computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in a processor of an electronic device, the processor in the electronic device executes the image reconstruction method.

[0114] Those skilled in the art can understand that all or some steps in the method disclosed above, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof. In the hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable storage medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media).

[0115] As those skilled in the art will appreciate, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable program instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as those skilled in the art will appreciate, communication media typically embodies computer readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. As used herein, the term "exemplary" means serving as an example, instance, or illustration. Any implementation described herein as exemplary is not necessarily to be construed as preferred or advantageous over other implementations.

[0116] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0117] Computer readable program instructions for carrying out operations of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or any combination of one or more of the above in any combination of one or more programming languages including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0118] The computer program product described herein can be embodied in a specific manner by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), and the like.

[0119] The various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.

[0120] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions which execute via the one or more processors of the computer or other programmable data processing devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0121] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0122] The flow and block diagrams in the drawings show architectural, functional, and operational aspects of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of instructions which comprise one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may

[0123] Example embodiments have been disclosed and, although a specific terminology is employed, it is merely for the convenience of the reader and is not intended to be limiting. In some instances, specific details have been included for the purpose of providing a thorough understanding of the embodiments presented, but it will be apparent to those skilled in the art that embodiments of the application can be practiced without these specific details. In some instances, features, characteristics and / or elements described in connection with a particular embodiment can be used in conjunction with, or switched to, features, characteristics and / or elements described in connection with other embodiments, unless otherwise explicitly stated. Accordingly, it will be understood that various changes in form and details can be made without departing from the scope of the disclosure as set forth in the appended claims.

Claims

1. An image reconstruction apparatus, characterized by comprising: The method comprises the following steps: a processing module is used to divide a target image into a plurality of sub-images and extract target region features contained in each sub-image; a global discrimination module is used to compare the target region features with preset rule region features for each sub-image; determine the image type of the sub-image according to the comparison result, and set a first loss weight corresponding to the sub-image according to the image type; wherein the first loss weight is used to represent the global feature loss of the sub-image; a local discrimination module is used to perform region feature discrimination processing on the target region features for each sub-image to obtain a region probability map corresponding to the sub-image; set a second loss weight corresponding to the sub-image according to the region probability map; wherein the second loss weight is used to represent the local feature loss of the sub-image; a reconstruction module is used to generate a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight corresponding to each sub-image respectively; and generate a reconstruction image corresponding to the target image according to the sub-reconstruction image corresponding to each sub-image.

2. The apparatus of claim 1, wherein, The similarity comparison is used to obtain a region feature similarity corresponding to the sub-image; The global discrimination module is specifically used to determine the image type of the sub-image according to the region feature similarity and a preset region feature similarity threshold; determine the first loss weight based on the region feature similarity and the image type; wherein the image type of the sub-image includes a rule type and a non-rule type, and the first loss weight of the sub-image of the rule type is greater than the first loss weight of the non-rule type.

3. The apparatus of claim 1 or 2, wherein, The local discrimination module is specifically used to: divide the sub-image into a plurality of local blocks; wherein the area of any local block is smaller than the area of the sub-image, and all extracted local blocks can be combined into the sub-image; for each local block, obtain a target local region feature corresponding to the local block; and perform similarity comparison on the target local region feature and a preset local region feature to generate a local region feature similarity corresponding to the local block; wherein the local region feature similarity is represented by a probability value; obtain the local region feature similarity of all local blocks, perform probability map conversion processing to obtain a region probability map corresponding to the sub-image.

4. The apparatus of claim 3, wherein, The local discrimination module is further used to: construct an initial probability matrix corresponding to the sub-image for the local region feature similarity of all local blocks; smooth the initial probability matrix to obtain a smoothed probability matrix corresponding to the initial probability matrix; determine the region probability map corresponding to the sub-image according to the smoothed probability matrix; the region probability map is represented by the smoothed probability matrix.

5. The apparatus of claim 1 or 2, 4, wherein, The reconstruction module is further used to: process each sub-image and the sub-reconstruction image corresponding to each sub-image to generate a first loss corresponding to the first loss weight, and generate a second loss corresponding to the second loss weight; generate a reconstruction loss value of each sub-reconstruction image according to the first loss weight and the first loss corresponding to the first loss weight, and the second loss weight and the second loss corresponding to the second loss weight; perform a correction process on the sub-reconstruction image according to the reconstruction loss value of each sub-reconstruction image, and generate a corrected reconstruction image according to the corrected sub-reconstruction image.

6. An image reconstruction method characterized by, comprising: divide a target image into a plurality of sub-images, and extract a target region feature contained in each sub-image; for each sub-image, compare the target region feature with a preset rule region feature in similarity; determine an image type of the sub-image according to a comparison result, and set a first loss weight corresponding to the sub-image according to the image type; wherein the first loss weight is used to represent a global feature loss of the sub-image; for each sub-image, perform a region feature discrimination process on the target region feature to obtain a region probability map corresponding to the sub-image; set a second loss weight corresponding to the sub-image according to the region probability map; wherein the second loss weight is used to represent a local feature loss of the sub-image; generate a sub-reconstruction image corresponding to each sub-image according to the first loss weight and the second loss weight corresponding to each sub-image respectively, and generate a reconstruction image corresponding to the target image according to the sub-reconstruction image corresponding to each sub-image.

7. The method of claim 6, wherein, The region feature discrimination process on the target region feature to obtain a region probability map corresponding to the sub-image includes: divide the sub-image into a plurality of local blocks; wherein the area of any local block is smaller than the sub-image, and all extracted local blocks can be combined into the sub-image; for each local block, obtain a target local region feature corresponding to the local block, and compare the target local region feature with a preset local region feature in similarity to generate a local region feature similarity corresponding to the local block; wherein the local region feature similarity is represented by a probability value; obtain the local region feature similarity of all local blocks, and perform a probability map conversion process to obtain a region probability map corresponding to the sub-image.

8. The method of claim 7, wherein, The obtaining of the local region feature similarity of all local blocks and the performing of the probability map conversion process to obtain the region probability map corresponding to the sub-image include: for the local region feature similarity of all local blocks, construct an initial probability matrix corresponding to the sub-image; smooth the initial probability matrix to obtain a smoothed probability matrix corresponding to the initial probability matrix; determine a region probability map corresponding to the sub-image according to the smoothed probability matrix; the region probability map is represented by the smoothed probability matrix.

9. An electronic device, comprising: The image reconstruction device comprises any one of claims 1-5.

10. An electronic device, comprising: comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein, The memory stores one or more computer programs that are executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 6-8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements the method of any one of claims 6-8.

Images