Image processing model training method and device, image processing method and device and electronic equipment

Abstract

The invention relates to an image processing model training method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of performing super-division processing on initial image data based on a to-be-trained generator model to obtain super-division image data; the resolution of the super-divided image data is greater than that of the initial image data; determining the depth loss of the to-be-trained generator model according to the depth difference between the super-division image data and the initial image data; and according to the depth loss, adjusting the to-be-trained generator model to obtain a target generator model. By adopting the method, complex depth information such as complex shadow and depth of field of image data can be reserved through the target generator model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing model training and image processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the continuous development of image processing technology, deep learning neural network models have become the mainstream image processing technology.

[0003] However, in the case of photography with mobile terminals such as mobile phones, the neural network model needs to be deployed and run on the terminal side. Due to limitations such as the time-consuming model inference and the number of parameters, the model's learning ability is limited. Therefore, when processing image data with complex depth information, it is difficult to restore natural details. Summary of the invention

[0004] The embodiments of the present application provide an image processing model training and image processing method, device, electronic device, and computer-readable storage medium, which can retain complex depth information such as complex light and shadow, depth of field, etc. of image data through a target generator model.

[0005] In a first aspect, the present application provides a method for training an image processing model, the method comprising:

[0006] Based on the generator model to be trained, super-resolution processing is performed on the initial image data to obtain super-resolution image data; the resolution of the super-resolution image data is greater than the resolution of the initial image data;

[0007] Determining a depth loss of the generator model to be trained according to a depth difference between the super-resolved image data and the initial image data;

[0008] According to the depth loss, the generator model to be trained is adjusted to obtain a target generator model.

[0009] In a second aspect, the present application also provides a training device for an image processing model, the device comprising:

[0010] A super-resolution module, used for performing super-resolution processing on the initial image data based on the generator model to be trained to obtain super-resolution image data; the resolution of the super-resolution image data is greater than the resolution of the initial image data;

[0011] A loss module, configured to determine a depth loss of the generator model to be trained according to a depth difference between the super-resolved image data and the initial image data;

[0012] A training module is used to adjust the generator model to be trained according to the depth loss to obtain a target generator model.

[0013] In a third aspect, the present application provides an image processing method, comprising:

[0014] Obtaining image data to be adjusted;

[0015] The image data to be adjusted is super-resolution processed through a target generator model to obtain target image data; wherein the target generator model is a generator model trained by the training method of the image processing model of the first aspect.

[0016] In a fourth aspect, the present application provides an image processing device, the device comprising:

[0017] An acquisition module, used for acquiring image data to be adjusted;

[0018] The super-resolution module is used to perform super-resolution processing on the image data to be adjusted through a target generator model to obtain target image data; wherein the target generator model is a generator model trained by the training device of the image processing model of the second aspect.

[0019] In a fifth aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the training method of the image processing model of the first aspect, or implements the image processing method of the third aspect.

[0020] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the image processing model of the first aspect, or implements the image processing method of the third aspect.

[0021] In a seventh aspect, the present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the training method of the image processing model of the first aspect, or implements the image processing method of the third aspect.

[0022] The training method of the above-mentioned image processing model performs super-resolution processing on the initial image data based on the generator model to be trained to obtain super-resolution image data, so as to reflect the result of the super-resolution processing of the generator model to be trained through the initial image data; and because the depth difference between the super-resolution image data and the initial image data can reflect the depth change before and after the super-resolution processing of the generator model to be trained, the depth loss of the generator model to be trained is determined according to the depth difference, and the generator model to be trained is adjusted according to the depth loss to obtain the target generator model. The target image data output by the target generator model is obtained by super-resolution processing, and due to the constraint of the depth loss, the light and shadow details of the initial image data are retained.

[0023] Accordingly, in the above-mentioned image processing method, since the target generator model is trained based on depth difference, it can more accurately retain depth information to produce a more natural light and shadow transition and a better depth of field effect, so that the generated image is closer to the original image or real scene in terms of light and shadow perception, thereby enhancing the realism of the output image and solving the problem of spatial and stereoscopic loss in traditional generative neural networks in image super-resolution tasks. As a result, the original spatial and stereoscopic senses are effectively prevented from being lost in the image super-resolution process, and more image details are retained. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A diagram of an application environment of a training method for an image processing model in one embodiment;

[0026] Figure 2 A schematic diagram of a flow chart of a method for training an image processing model in one embodiment;

[0027] Figure 3 A schematic diagram of a process for determining depth loss in one embodiment;

[0028] Figure 4 A schematic diagram of a process for determining a depth feature in one embodiment;

[0029] Figure 5 A schematic diagram of a process for determining a local depth feature in one embodiment;

[0030] Figure 6 A schematic diagram of a process for determining a global depth feature in one embodiment;

[0031] Figure 7 A schematic diagram of a process of training a generator model in one embodiment;

[0032] Figure 8 is a schematic flow chart of an image processing method in one embodiment;

[0033] Fig. 9 This is the effect diagram of the generative adversarial image super-resolution network in traditional technology;

[0034] Fig.10 A schematic diagram of a process of training and reasoning an image processing model in one embodiment;

[0035] Fig.11 A schematic diagram of a process flow of a depth estimation network processing in one embodiment;

[0036] Fig.12 is a structural block diagram of a training device for an image processing model in one embodiment;

[0037] Fig.13 is a structural block diagram of an image processing device in one embodiment;

[0038] Fig.14 is a diagram of the internal structure of an electronic device in one embodiment;

[0039] Fig.15 FIG. 4 is a diagram showing the internal structure of an electronic device in another embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] The training method of the image processing model provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0042] The electronic device 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0043] In some exemplary embodiments, Figure 2 As shown, a training method for an image processing model is provided, and the method is applied to Figure 1 The method is described using the server 104 as an example and can also be applied to the electronic device 102. The method includes the following steps 202 to 206.

[0044] Step 202: Based on the generator model to be trained, the initial image data is super-resolved to obtain super-resolved image data; the resolution of the super-resolved image data is greater than the resolution of the initial image data.

[0045] The generator model to be trained is a generator model that needs to be trained. The generator model to be trained can be an initial generator model before training, or it can be an adjusted generator model during the training process. The generator model to be trained is a neural network model, and the generator model to be trained can perform super-resolution processing based on the image data input to itself to adjust the resolution of the image data. Optionally, the generator model to be trained is a generative neural network model that can reconstruct higher-resolution super-resolution image data based on the image data input thereto.

[0046] Optionally, the generator model to be trained belongs to the generator model in the Generative Adversarial Networks (GAN), and correspondingly, the generator model to be trained and the discriminator model in the GAN can be trained alternately, so that the target generator model can perform super-resolution on the image data and better restore the depth of the image data.

[0047] The initial image data is image data with a resolution lower than a resolution threshold. The initial image data is part of the sample data in the model training process. The initial image data at least includes the pixel value of each pixel, which can reflect the depth information and color information of the initial image data. Optionally, the initial image data is data belonging to a low-resolution image (LR) in a generative adversarial network.

[0048] The super-resolution image data is the result of super-resolution processing by the generator model to be trained. Since the super-resolution processing can improve the resolution of the initial image data, the resolution of the super-resolution image data is higher than the resolution of the initial image data. Optionally, since the resolution of the super-resolution image data is greater than the resolution of the initial image data, each pixel in the initial image data corresponds to a plurality of pixels in the super-resolution image data. Optionally, when the corresponding initial image is displayed through the initial image data, and the corresponding super-resolution image is displayed through the super-resolution image data, the clarity of the initial image is higher than the clarity of the super-resolution image.

[0049] In some optional embodiments, based on the generator model to be trained, the initial image data is super-resolution processed to obtain super-resolution image data, including: up-sampling the initial image data to obtain first high-resolution image data; dividing the first high-resolution image data into multiple blocks of image data, and obtaining the characteristic value of each block of image data; determining the filter corresponding to each block of image data according to the characteristic value of each block of image data, and using each filter to filter the corresponding image data block in the first high-resolution image data to obtain second high-resolution image data; fusing the second high-resolution image data with the first high-resolution image data to obtain super-resolution image data. Thus, the image data with the missing part supplemented is obtained by up-sampling, and then the characteristic value of the block image data is calculated, so as to quickly query the trained filter corresponding to the block image data according to the characteristic value for filtering, and then perform fusion, so as to reduce the artificial traces between each image block in the super-resolution image data, maintain the structural information of the image and obtain a clear image.

[0050] In some optional embodiments, based on the generator model to be trained, the initial image data is super-resolved to obtain the super-resolved image data, including: based on the generator model with a U-shaped structure, the initial image data is super-resolved to obtain the super-resolved image data. Optionally, the processing method of the generator model with a U-shaped structure includes: inputting the initial image data into an encoder with a gradually decreasing spatial resolution and a gradually increasing feature dimension to obtain a high-level semantic feature map; then inputting the high-level semantic feature map into a decoder with a gradually increasing spatial resolution and a gradually decreasing feature dimension, and finally obtaining the super-resolved image data through an upsampling module. Therefore, since the encoder has the characteristics of gradually decreasing spatial resolution and gradually increasing feature dimension, noise reduction can be achieved, and since the decoder has the characteristics of gradually increasing spatial resolution and gradually decreasing feature dimension, the high-level semantic feature map after noise reduction can be restored at a higher scale. Finally, through upsampling, the super-resolved image data can reach the required resolution specification to ensure clarity.

[0051] In some optional embodiments, based on the generator model to be trained, the initial image data is super-resolution processed to obtain super-resolution image data, including: based on the generator model of the lightweight image super-resolution network SCNet with full point-by-point convolution, the initial image data is super-resolution processed to obtain super-resolution image data.

[0052] Step 204, determining the depth loss of the generator model to be trained according to the depth difference between the super-resolved image data and the initial image data.

[0053] The depth difference is the difference between the depth features of the super-resolved image data and the initial image data. Optionally, when the initial image data is collected by a depth camera, the initial image data contains depth information of its own pixel values, and the features extracted from the depth information are depth features. Optionally, when the depth of the initial image data is detected by a depth estimation network (Depth Estimation), the depth features of the initial image data can be obtained; when the depth of the super-resolved image data is detected by a depth estimation network, the depth features of the super-resolved image data can be obtained.

[0054] The depth difference can be a difference calculated based on the difference between the depth features of the super-resolved image data and the depth features of the initial image data; the depth difference can also be the difference between the absolute value of the depth features of the super-resolved image data and the absolute value of the depth features of the initial image data; the depth difference can also be a ratio of the depth feature distribution between the initial image data and the super-resolved image data.

[0055] The depth loss is used to evaluate the depth information retention of the generator model to be trained. Optionally, the depth loss can be obtained by substituting the depth difference into the loss function for adjustment; when the depth loss is the depth difference, it can be evaluated more efficiently; when the depth loss is obtained by adjusting the depth difference based on the loss function, the depth loss can be determined more accurately. Optionally, the loss function can be a loss function of regularization terms such as L1 distance, L2 distance, etc.; the loss function can also be a loss function for calculating the degree of difference, such as KL divergence, Bhattacharyya distance, chi-square, etc. The depth loss is determined based on the depth difference at at least one scale.

[0056] In some optional embodiments, the depth loss of the generator model to be trained is determined according to the depth difference between the super-resolved image data and the initial image data, including: extracting the depth features of the super-resolved image data through a depth estimation network, and extracting the depth features of the initial image data through a depth estimation network; performing difference calculation on the depth features of the super-resolved image data and the depth features of the initial image data to obtain the depth loss of the generator model to be trained. Wherein, the depth estimation network includes but is not limited to types such as monocular depth estimation (Monocular Depth Estimation) and binocular depth estimation (StereoDepth Estimation); wherein the monocular depth estimation network has a faster reasoning speed, fewer parameters and higher depth accuracy, and is more suitable as a supervisory network for the generator model to be trained. For example: Depth Anything V2 is a basic model for monocular depth estimation with SOTA accuracy; compared with the model based on stable diffusion (SD), Depth Anything V2 has a faster reasoning speed, fewer parameters and higher depth accuracy, and is more suitable as a supervisory network for the generator model to be trained.

[0057] Step 206: According to the depth loss, the generator model to be trained is adjusted to obtain a target generator model.

[0058] The target generator model is a generator model after the generator model to be trained has been adjusted at least once. The target generator model can perform super-resolution processing based on the image data input to itself to adjust the resolution of the image data, and enable the super-resolution image data to retain the depth information of the image data, and can better restore the depth information of the image data to be adjusted. Optionally, the target generator model is obtained by adjusting the model weights of the generator model to be trained based on a generative adversarial network.

[0059] In some optional embodiments, the generator model to be trained is adjusted according to the depth loss to obtain the target generator model, including: performing gradient descent processing according to the depth loss to obtain the gradient of each parameter adjustment contained in the loss function; updating the model weights of the generator model to be trained according to the gradient until the stop training condition corresponding to the generator model to be trained is met, and the target generator model is obtained. Thus, the model weights of the generator model to be trained are executed separately through the gradient descent algorithm to train accurately and efficiently.

[0060] In some optional embodiments, the generator model to be trained is adjusted according to the depth loss to obtain a target generator model, including: calculating the gradient of each layer layer by layer from the output layer of the generator model to be trained to the input layer of the generator model to be trained according to the depth loss; updating the weight of each layer according to the gradient of each layer to obtain an adjusted generator model; and obtaining a target generator model according to the adjusted generator model that meets preset conditions. Thus, the model weights of the generator model to be trained are executed separately through an algorithm for calculating the gradient layer by layer to train more accurately.

[0061] In some optional embodiments, a generative adversarial network is used for training, and the training phase of the generative adversarial network includes the training of a generator to be trained and a discriminator. The generator to be trained is used to generate super-resolution image data that is closer and closer to the reference image data to deceive the discriminator, while the discriminator improves its ability to distinguish whether the super-resolution image data is the reference image data. Through the two models that are in confrontation with each other, the two models are continuously adjusted and optimized to finally achieve the training goal; the training goal is to generate a high-resolution image close to the reference image data (ground truth, GT) through the target generator model.

[0062] Optionally, the generator to be trained is a U-net that can be deployed on the end. The initial image data input to the generator to be trained is passed through an encoder with gradually decreasing spatial resolution and gradually increasing feature dimensions to obtain a high-level semantic feature map; the high-level semantic feature map is then passed through a decoder with gradually increasing spatial resolution and gradually decreasing feature dimensions, and finally through an upsampling module to obtain super-resolution image data. Correspondingly, the discriminator is another U-net with a smaller size than the generator. It is similar to the generator, but has no upsampling module. It inputs super-resolution image data or reference image data, and outputs a tensor with the same spatial resolution as the input and the number of channels as 1. This tensor represents the confidence of the discriminator in judging that the input image is True in the pixel dimension.

[0063] Optionally, the generator model to be trained is adjusted according to the depth loss to obtain the target generator model, including: in the first stage, the generator model to be trained is adjusted according to the depth loss to obtain the adjusted generator model; in the second stage, the corresponding discriminator to be trained is adjusted to obtain the adjusted discriminator network; in the third stage, the target generator model is determined according to the adjusted generator model based on the adjusted generator model and the adjusted discriminator network. Thus, the adjusted generator model and the adjusted discriminator model are obtained respectively, rather than directly conducting adversarial training, so as to avoid the unstable convergence problem in direct adversarial training. After the individual training of the generator model to be trained is completed, the super-resolution image data to be discriminated is obtained, and the discriminator model to be trained is trained by the super-resolution image data to be discriminated, so as to form a two-stage training of the generative adversarial network, so as to ensure that the image data output by the target generator model retains specific depth information. Moreover, the ability of the generator to retain depth information is enhanced by alternating training.

[0064] In the training method of the above-mentioned image processing model, based on the generator model to be trained, the initial image data is super-resolution processed to obtain super-resolution image data, so that the result of the super-resolution processing of the generator model to be trained can be reflected through the initial image data; and due to the depth difference between the super-resolution image data and the initial image data, the depth change before and after the super-resolution processing of the generator model to be trained can be reflected, so according to the depth difference, the depth loss of the generator model to be trained is determined, and according to the depth loss, the generator model to be trained is adjusted to obtain the target generator model. The target image data output by the target generator model is obtained by super-resolution processing, and due to the constraint of the depth loss, the light and shadow details of the initial image data are retained.

[0065] In one embodiment, Figure 3 As shown, according to the depth difference between the super-resolved image data and the initial image data, the depth loss of the generator model to be trained is determined, including steps 302 to 308. Wherein:

[0066] Step 302: Perform depth feature detection at multiple scales on the initial image data to obtain initial depth features at multiple scales.

[0067] The initial depth feature is a depth feature detected from the initial image data, and can reflect the depth information of the initial image data. Optionally, the initial depth feature can be generated based on the depth information contained in the initial image data, or can be extracted from the initial image data.

[0068] Optionally, since the resolution of the initial image data is lower than that of the super-resolution image data, the initial image data may be upsampled, and then depth feature detection at multiple scales may be performed based on the upsampled initial image data to obtain initial depth features at multiple scales. Optionally, since the resolution of the initial image data is lower than that of the super-resolution image data, fusion may be performed based on the initial depth features at multiple scales, so that the initial depth features at multiple scales and the super-resolution depth features at multiple scales have one-to-one corresponding pixel positions.

[0069] Optionally, the initial depth features of multiple scales can reflect the depth information of the initial image data at different precisions. Optionally, based on the range corresponding to each scale, a depth feature within each range can be determined. Optionally, the multiple scales include at least two of the pixel scale, the local scale and the global scale, wherein the initial depth feature at the pixel scale includes the initial depth feature of each pixel, i.e., the initial pixel depth feature; the initial depth feature at the local scale, i.e., the initial local depth feature, which is obtained based on the statistics of the initial pixel depth feature of each block of pixels, and each block of pixels contains multiple pixels; the initial depth feature at the global scale, i.e., the initial global depth feature, which is obtained based on the statistics of the initial pixel depth feature of each pixel.

[0070] In some optional embodiments, the initial image data is subjected to depth feature detection at multiple scales to obtain initial depth features at multiple scales, including: dividing the initial image data into multiple regions of different scales, extracting depth features from each region of the initial image data, and combining the depth features of each region of the initial image data to obtain initial depth features at multiple scales. Thus, feature extraction is performed at multiple scales respectively without fusion to decouple the depth features at multiple scales, so that the loss function calculation process at the corresponding scale is more independent and easier to adjust.

[0071] In some optional embodiments, the initial image data is subjected to depth feature detection at multiple scales to obtain initial depth features at multiple scales, including: extracting depth features from the initial image data to obtain a depth feature map of the initial image data; performing multiple rounds of convolution and pooling processing on the depth feature map of the initial image data to obtain an initial depth feature map with a gradually decreasing resolution; performing multiple rounds of upsampling processing based on the initial depth feature map with a gradually decreasing resolution to obtain an initial upsampled depth feature map with a gradually increasing resolution; performing multi-scale feature fusion on the initial depth feature map obtained by each round of convolution and pooling processing and the initial upsampled depth feature map obtained by each round of upsampling processing to obtain initial depth features at multiple scales. Thus, multi-scale feature fusion is performed through the image pyramid model to obtain depth features with scales fused together, thereby ensuring the accuracy of the features.

[0072] Step 304: perform depth feature detection at multiple scales on the super-resolved image data to obtain super-resolved depth features at multiple scales.

[0073] The super-resolution depth feature is the depth feature of the super-resolution image data. The scale of the super-resolution depth feature and the initial depth feature has a corresponding number to perform depth feature fusion of the same scale.

[0074] The super-resolution depth feature is a depth feature extracted from the super-resolution image data, which can reflect the depth information of the super-resolution image data. Optionally, the super-resolution depth features of multiple scales can reflect the depth information of the super-resolution image data at different precisions. Optionally, based on the range corresponding to each scale, a depth feature within each range can be determined. Optionally, the multiple scales include at least two of the pixel scale, the local scale and the global scale, wherein the super-resolution depth feature at the pixel scale includes the super-resolution depth feature of each pixel.

[0075] In some optional embodiments, multiple scales of depth feature detection are performed on the super-resolution image data to obtain multiple scales of super-resolution depth features, including: dividing the super-resolution image data into regions of multiple scales, and extracting depth features from each region of the super-resolution image data; combining the depth features of each super-resolution image data in the region to obtain multiple scales of super-resolution depth features. Thus, feature extraction is performed separately at multiple scales without fusion to decouple the depth features of multiple scales, so that the loss function calculation process at the corresponding scale is more independent and easy to adjust.

[0076] In some optional embodiments, multiple scales of depth feature detection are performed on the super-resolution image data to obtain multiple scales of super-resolution depth features, including: extracting depth features from the super-resolution image data to obtain a depth feature map of the super-resolution image data; performing multiple rounds of convolution and pooling on the depth feature map of the super-resolution image data to obtain a super-resolution depth feature map with a gradually decreasing resolution; performing multiple rounds of upsampling based on the super-resolution depth feature map with a gradually decreasing resolution to obtain a super-resolution up-sampled depth feature map with a gradually increasing resolution; performing multi-scale feature fusion on the super-resolution depth feature map obtained by each round of convolution and pooling and the super-resolution up-sampled depth feature map obtained by each round of upsampling to obtain multiple scales of super-resolution depth features. Thus, multi-scale feature fusion is performed through the image pyramid model to obtain depth features with scales fused together, thereby ensuring the accuracy of the features.

[0077] Step 306, for each scale, based on the difference between the super-resolved depth feature of the first pixel and the super-resolved depth feature of the second pixel, determine the depth loss of each scale; the first pixel in the initial image data of the same scale is matched with the second pixel in the super-resolved image data.

[0078] The first pixel point includes at least one pixel point in the initial image data, and the depth feature extracted from the pixel value of the first pixel point is the depth feature of the first pixel point. At different scales, the number of pixels contained in the first pixel point is different; at the pixel scale, the first pixel point includes each independent pixel point; at the regional scale, the first pixel point includes multiple pixels in the region corresponding to the regional scale. Optionally, the regional scale includes at least one scale of a local scale and a global scale, wherein, in the local scale, the first pixel point includes each block of pixels, and each block of pixels includes multiple pixels; in the global scale, the first pixel point includes each pixel.

[0079] The second pixel point includes at least one pixel point in the super-resolution image data, and the depth feature extracted from the pixel value of the second pixel point is the depth feature of the second pixel point. At different scales, the number of pixels contained in the second pixel point is different; at the pixel scale, the second pixel point includes each independent pixel point; at the regional scale, the second pixel point includes multiple pixels in the region corresponding to the regional scale. Optionally, the regional scale includes at least one scale of a local scale and a global scale; wherein, within the local scale, the second pixel point includes each block of pixels, and each block of pixels includes multiple pixels; within the global scale, the second pixel point includes each pixel.

[0080] The second pixel point matches the first pixel point at the same scale; at the same scale, the depth features of the first pixel point and the second pixel point have matching accuracy. Optionally, for the second pixel point matching the first pixel point, this situation refers to the situation where the pixel positions correspond, and the situation where the pixel positions correspond includes but is not limited to the same pixel position, a deviation position with a preset deviation value from the pixel position, and a pixel position with feature matching.

[0081] Optionally, the pixel coordinates of the first pixel point in the initial image data are (x, y); and at the same scale, the pixel coordinates of the second pixel point in the super-resolved image data are (x, y); or, the pixel coordinates of the second pixel point in the super-resolved image data are (x±a, y±a), where a is a preset deviation value; or, the pixel coordinates of the second pixel point in the super-resolved image data are (j, k), where j is the coordinate point obtained based on the x mapping, and k is the coordinate point obtained based on the y mapping.

[0082] In some optional embodiments, for each scale, based on the difference between the super-resolution depth feature of the first pixel and the super-resolution depth feature of the second pixel, the depth loss of each scale is determined, including: according to the loss function corresponding to each scale, respectively calculating the difference between the super-resolution depth feature of the first pixel and the super-resolution depth feature of the second pixel, to obtain the depth loss of each scale. Thus, the loss function at the corresponding scale is more independent and easy to adjust.

[0083] In some optional embodiments, for each scale, based on the difference between the super-resolution depth feature of the first pixel and the super-resolution depth feature of the second pixel, the depth loss of each scale is determined, including: at each scale, according to a preset loss function, the difference between the super-resolution depth feature of the first pixel and the super-resolution depth feature of the second pixel is determined to obtain the depth loss of each scale. Thus, the loss functions at the corresponding scales are all preset to ensure processing efficiency.

[0084] Step 308: Determine the depth loss of the generator model to be trained according to the depth loss of each scale.

[0085] In some optional embodiments, the depth loss of the generator model to be trained is determined according to the depth loss of each scale, including: fusing the depth loss of each scale, and obtaining the depth loss of the generator model to be trained according to the fusion result. The fusion method includes but is not limited to accumulation and multiplication. Thus, the fusion is directly performed to ensure processing efficiency.

[0086] In some optional embodiments, the depth loss of the generator model to be trained is determined according to the depth loss of each scale, including: weighting the depth loss of each scale to obtain the weighted depth loss of each scale; fusing the weighted depth loss of each scale to obtain the depth loss of the generator model to be trained; wherein the fusion method includes but is not limited to accumulation and multiplication. Thus, weighting is performed first and then fusion is performed, taking into account both processing efficiency and accuracy.

[0087] In this embodiment, since the depth features of different scales are the depth information of the image data in different ranges, the depth loss of each scale is obtained by the depth difference between the depth features of each scale, which can reflect the depth changes of the super-resolved image data at different levels and angles; therefore, the depth loss of the generator model to be trained is determined by combining the depth losses of each scale, so that the depth features of the super-resolved image data output by the generator model to be trained can gradually approach the depth features of the initial image data.

[0088] In some optional embodiments, the initial depth features of multiple scales include initial pixel depth features and initial regional depth features, and the super-resolution depth features of multiple scales include super-resolution pixel depth features and super-resolution regional depth features. The initial pixel depth feature and the super-resolution pixel depth feature belong to the depth features of the pixel scale, and the initial regional depth feature and the super-resolution regional depth feature belong to the depth features of the regional scale. That is, multiple scales include pixel scale and regional scale, and the pixel scale and regional scale distribution have their own initial depth features and super-resolution depth features. Among them, the pixel scale is the scale composed of each pixel, and each pixel has its own depth feature at the pixel scale, thereby forming a depth feature at the pixel granularity. The regional granularity is the scale composed of each pixel, and each pixel has its own depth feature at the pixel scale, thereby forming a depth feature at the regional granularity.

[0089] like Figure 4 As shown, the initial image data is subjected to depth feature detection at multiple scales to obtain initial depth features at multiple scales, including steps 402 and 404; correspondingly, the super-resolved image data is subjected to depth feature detection at multiple scales to obtain super-resolved depth features at multiple scales, including steps 406 and 408, wherein:

[0090] Step 402: Perform depth detection on the initial image data to obtain initial pixel depth features.

[0091] In some optional embodiments, depth detection is performed on the initial image data to obtain initial pixel depth features, including: performing depth detection on the initial image data through a monocular depth estimation network to obtain initial pixel depth features.

[0092] In some optional embodiments, performing depth detection on the initial image data to obtain the initial pixel depth feature includes: combining based on the depth information contained in the initial image data to obtain the initial pixel depth feature. Thus, when the initial image data itself contains depth information, the depth information contained therein is directly used to ensure the feature extraction speed.

[0093] Step 404: Determine the initial region depth feature based on the feature distribution of the initial pixel depth feature.

[0094] In some optional embodiments, the initial regional depth feature is determined based on the feature distribution of the initial pixel depth feature, including: dividing the initial pixel depth feature into each region according to a preset number of pixels to obtain the initial pixel depth feature of each region; performing statistics based on the initial pixel depth feature of each region to obtain the initial regional depth feature. The statistical method may be, but is not limited to, a method based on a histogram, variance or standard deviation. Thus, by dividing the blocks according to the number of pixels, the depth features can be carefully counted for the corresponding resolution.

[0095] In some optional embodiments, the initial regional depth features are determined based on the feature distribution of the initial pixel depth features, including: based on the pixel position in the depth map composed of the initial pixel depth features, each pixel position within a preset range is determined as each region; the initial pixel depth features in each region are data fused to obtain the initial pixel depth features in each region; the initial pixel depth features in each region belong to the initial regional depth features. The data fusion method can be but is not limited to a method based on the mean, variance or standard deviation. Therefore, the blocks are divided according to the length, which is more versatile.

[0096] Step 406: perform depth detection on the super-resolved image data to obtain super-resolved pixel depth features.

[0097] In some optional embodiments, depth detection is performed on the super-resolved image data to obtain super-resolved pixel depth features, including: performing depth detection on the super-resolved image data through a monocular depth estimation network to obtain super-resolved pixel depth features.

[0098] In some optional embodiments, depth detection is performed on the super-resolution image data to obtain super-resolution pixel depth features, including: based on the Depth Anything V2 model, depth detection is performed on the super-resolution image data to obtain super-resolution pixel depth features. The Depth Anything V2 model is a monocular depth estimation network with SOTA accuracy.

[0099] Step 408: Determine the super-resolved region depth feature based on the feature distribution of the super-resolved pixel depth feature.

[0100] In some optional embodiments, based on the feature distribution of the super-resolution pixel depth feature, the super-resolution regional depth feature is determined, including: dividing the super-resolution pixel depth feature into each region according to the number of pixels corresponding to each region, and obtaining the super-resolution pixel depth feature of each region; performing statistics based on the super-resolution pixel depth feature of each region to obtain the super-resolution regional depth feature. Among them, the statistical method can be but is not limited to a method based on a histogram, variance or standard deviation. Therefore, by dividing the blocks according to the number of pixels, the depth features can be carefully counted for the corresponding resolution.

[0101] In some optional embodiments, based on the feature distribution of the super-resolution pixel depth feature, the super-resolution regional depth feature is determined, including: based on the pixel position in the depth map composed of the super-resolution pixel depth feature, each pixel position within a preset range is determined as each region; the super-resolution pixel depth feature in each region is data fused to obtain the super-resolution pixel depth feature in each region; the super-resolution pixel depth feature in each region belongs to the super-resolution regional depth feature. Among them, the data fusion method can be but is not limited to a method based on the mean, variance or standard deviation. Therefore, the block is divided according to the length, which is more universal.

[0102] In this embodiment, the initial pixel depth feature and the super-resolution pixel depth feature are first determined to form the depth feature of the pixel granularity; and the initial regional depth feature and the super-resolution regional depth feature of the regional granularity are determined by the statistics of feature distribution to reflect the local depth information common to multiple pixels. Thus, the depth features at different scales are formed by the pixel granularity and the regional granularity, so that there are corresponding depth differences at the pixel scale and the regional scale, so that when the generator model to be trained is super-resolution processed, richer depth features can be retained.

[0103] In some optional embodiments, the initial pixel depth features and the super-resolution pixel depth features are detected by a depth estimation network; the weights of the depth estimation network are fixed, or the weights of the depth estimation network are updateable.

[0104] For the weights of the depth estimation network, the update process is determined based on the preset tasks and loss functions, which include but are not limited to: mean squared error (MSE), mean absolute error (MAE), Huber loss (Huber Loss), improved Huber loss (berHu Loss), binary cross entropy loss (BCE Loss), and weighted cross entropy loss (Weighted Cross Entropy Loss, Weighted CE Loss).

[0105] Optionally, the network weights loaded by the depth estimation network used are open source pre-trained weights, which are trained based on public and general data, so the weights of the depth estimation network are fixed.

[0106] Optionally, in view of the distribution of data in the custom task, in order to further improve the fidelity of depth estimation to image structure light and shadow during training and reduce the differences between data domains, it can be jointly learned with the generator and the discriminator, that is, after obtaining the adjusted generator model trained separately and completing the first stage, the weights of the depth estimation network are unfixed, so that the weights of the depth estimation network are updateable, and the depth features output by the depth estimation network are more accurate and more precise, which ultimately makes the target image data output by the target generator more realistic.

[0107] In this embodiment, when the weights of the depth estimation network are fixed, the model training speed can be improved; and when the weights of the depth estimation network are updateable, the fidelity of the depth estimation to the image structure light and shadow during training can be improved, and the difference between data domains can be reduced, making the output image depth more accurate and more precise, ultimately enabling the target image data super-resolved by the target generator to display a more realistic image.

[0108] In some optional embodiments, depth detection is performed on the initial image data to obtain initial pixel depth features, including: performing depth detection on the initial image data to obtain depth features to be sampled of the initial image data; and upsampling the depth features to be sampled according to the size of the super-resolution image data to obtain initial pixel depth features.

[0109] The depth feature to be sampled is a depth feature determined based on the initial image data. The depth feature to be sampled corresponds to the scale of the initial image data. Optionally, the depth feature to be sampled can be a depth feature output by a monocular depth estimation network after depth detection is performed on the initial image data through a monocular depth estimation network.

[0110] Optionally, at least one of a neural network and an interpolation method may be used to upsample the depth features to be sampled. Optionally, the neural network method uses transposed convolution to insert a preset value (such as 0) into the feature map composed of the depth features to be sampled to fill it, and then apply a normal convolution operation. In this way, the size of the feature map can be effectively enlarged. Optionally, upsampling can be performed based on bilinear interpolation, nearest neighbor interpolation, bicubic upsampling, etc.

[0111] In some optional embodiments, taking bicubic interpolation upsampling as an example, the depth feature to be sampled is upsampled according to the size of the super-resolution image data to obtain an initial pixel depth feature, including: determining the number of interpolation points in the horizontal and vertical directions according to the size difference between the super-resolution image data and the size of the depth feature to be sampled; in the depth feature to be sampled, dividing the neighborhood pixel set of the interpolation point according to the number of interpolation points; mapping the pixel values ​​contained in the neighborhood pixels whose pixel spacing with the interpolation point is less than the first pixel threshold according to a cubic interpolation function positively correlated with the pixel spacing to obtain a first mapping value; mapping the pixel values ​​contained in the neighborhood pixels whose pixel spacing with the interpolation point is greater than the first pixel threshold and less than the second pixel threshold according to a quadratic interpolation function positively correlated with the pixel spacing to obtain a second mapping value; performing weighted processing according to the first mapping value and the second mapping value to obtain the pixel value of the interpolation point; the pixel value of the interpolation point is obtained by interpolation operation, the depth feature to be sampled includes the pixel value of the neighborhood pixel set, and does not include the pixel value of the interpolation point; the initial pixel depth feature includes the pixel value of the interpolation point and the pixel value of the depth feature to be sampled.

[0112] Thus, the first mapping value is obtained to ensure that the weight affecting the pixel is the largest, thereby ensuring the smoothness of the interpolation; the second mapping value is obtained to ensure that the weight affecting the pixel begins to decrease until it decreases to 0. This transition helps to further enhance the smoothness of the interpolation and ensure that no obvious jump occurs.

[0113] Optionally, the pixel values ​​contained in the neighboring pixels whose pixel distance to the interpolation point is greater than the second pixel threshold will no longer affect the interpolation result, which helps to maintain the accuracy of the interpolation and avoid the computational burden. Optionally, the pixel values ​​contained in the neighboring pixels whose pixel distance to the interpolation point is greater than the second pixel threshold are adjusted using other methods to the interpolation result to ensure the image accuracy in the corresponding scene.

[0114] In this embodiment, the depth detection is first performed on the initial image data, and then up-sampling is performed based on the depth features to be sampled obtained by the depth detection, which can avoid the up-sampling process interfering with the depth detection process of the initial image data, making the initial pixel depth features more accurate. Moreover, according to the size of the super-resolution image data, the up-sampling process of the depth features to be sampled can make the initial pixel depth features and the super-resolution pixel depth features have matching resolutions, so as to facilitate the calculation of depth differences.

[0115] In some optional embodiments, the initial regional depth feature includes the initial local depth feature, and the super-resolution regional depth feature includes the super-resolution local depth feature; in this case, the regional scale includes the local scale, and the depth feature at the local scale includes the initial local depth feature and the super-resolution local depth feature; the local scale includes the common depth feature of each block of pixels, and each block of pixels includes multiple pixels. For example: within the local scale, the first pixel point includes each pixel block of the feature map composed of the initial pixel depth feature; within the local scale, the second pixel point includes each pixel block of the feature map composed of the super-resolution pixel depth feature.

[0116] like Figure 5 As shown, based on the feature distribution of the initial pixel depth feature, the initial region depth feature is determined, including steps 502 and 504; correspondingly, based on the feature distribution of the super-resolution pixel depth feature, the super-resolution region depth feature is determined, including steps 506 and 508, wherein:

[0117] Step 502 , divide the depth map formed by the initial pixel depth features into blocks to obtain the initial pixel depth features of each block.

[0118] The depth map is a set of depth features at the pixel scale. The pixel values ​​of the depth map are depth values, which are used to represent the depth information of the initial depth features at the pixel scale at the same pixel position. The pixel positions of the depth map composed of the initial pixel depth features correspond to the pixel positions in the initial image data, and the pixel positions of the depth map composed of the super-resolution pixel depth features correspond to the pixel positions in the super-resolution image data.

[0119] Optionally, the depth map formed by the initial pixel depth features corresponds to the pixel positions in the initial image data one-to-one, and each pixel position in the depth map has a depth value that is used to represent the depth information of the initial image data at the same pixel position; optionally, the depth map formed by the initial pixel depth features corresponds to at least one pixel position in the initial image data, and each pixel position in the depth map has a depth value that is used to represent the depth information of the initial image data at the corresponding at least one pixel position.

[0120] Optionally, the depth map composed of the super-resolution pixel depth features corresponds to the pixel positions in the super-resolution image data one-to-one, and each pixel position in the depth map has a depth value, which is used to represent the depth information of the super-resolution image data at the same pixel position; optionally, the depth map composed of the super-resolution pixel depth features corresponds to multiple pixel positions in the super-resolution image data, and each pixel position in the depth map has a depth value, which is used to represent the depth information of the super-resolution image data at the corresponding multiple pixel positions.

[0121] The initial pixel depth features of each block include the initial depth features of the initial image data at the pixel position of each block. Optionally, the depth map formed by the initial pixel depth features is divided into M blocks, and the M-block depth map contains N pixels, then there are M blocks of initial pixel depth features, and each block has N initial pixel depth features; M and N are positive integers respectively.

[0122] In some optional embodiments, the depth map composed of the initial pixel depth features is divided into blocks to obtain the initial pixel depth features of each block, including: dividing the depth map composed of the initial pixel depth features into blocks according to a preset size to obtain the initial depth features in each preset size block. The preset size may be at least one of length, width, area or number of pixels. For example: using a 64*64 block, a depth map has a total of 4*4 image blocks, and the initial pixel depth features of 4*4 blocks are obtained.

[0123] In some optional embodiments, dividing the depth map formed by the initial pixel depth features into blocks to obtain the initial pixel depth features of each block includes: dividing the depth map formed by the initial pixel depth features into blocks according to preset intervals to obtain the initial pixel depth features between each preset interval. The preset interval may be at least one of a length, a width, an area, or a number of pixels.

[0124] Step 504 , performing distribution statistics on the depth values ​​of the initial pixel depth features of each block to obtain an initial local depth feature.

[0125] The initial local depth feature represents the depth value distribution information of the initial pixel depth feature of each block. The depth value distribution information can represent the degree of change of the depth value of each block to reflect the depth information at the local scale in detail. Optionally, in the case where part of the light and shadow of the initial image data is only located in a certain image block of its depth map, the depth information of this part of the light and shadow can be reflected through the initial local depth feature. Optionally, the initial local depth feature includes but is not limited to: the variance, standard deviation, mean square error or correlation coefficient of the depth value.

[0126] Optionally, there are initial pixel depth features of M blocks, and each block has N initial pixel depth features. Then, based on the N initial pixel depth features of each block, the initial pixel depth features of the M blocks are determined, and the initial pixel depth features of the M blocks are used as the initial local depth features.

[0127] In some optional embodiments, the depth values ​​of the initial pixel depth features of each block are distributed and counted to obtain the initial local depth features, including: performing variance calculation based on the depth values ​​of the initial pixel depth features of each block to obtain the initial pixel depth feature variance of each block; using the initial pixel depth feature variance of each block as the initial local depth feature. Optionally, the size of the initial pixel depth feature variance is positively correlated with the degree of change in the depth value of the corresponding area; if the variance is smaller, it means that the depth value of the area does not change much, that is, the depth is more uniform; if the variance is larger, it means that the depth value of the area changes more, and there may be obvious changes in texture or edges.

[0128] Step 506 , dividing the depth map formed by the super-resolution pixel depth features into blocks to obtain super-resolution pixel depth features of each block.

[0129] The super-resolution pixel depth features of each block include super-resolution pixel depth features of the super-resolution image data at the pixel position of each block. Optionally, the depth map formed by the super-resolution pixel depth features is divided into M blocks, and the M-block depth map contains N pixels, then there are super-resolution pixel depth features of M blocks, and each block has N super-resolution pixel depth features.

[0130] In some optional embodiments, the depth map composed of the super-resolution pixel depth features is divided into blocks to obtain the super-resolution pixel depth features of each block, including: dividing the depth map composed of the super-resolution pixel depth features into blocks according to a preset size to obtain the super-resolution pixel depth features within each preset size. The preset size may be at least one of length, width, area or number of pixels. For example: using a 64*64 block, a depth map has a total of 4*4 image blocks, and the super-resolution pixel depth features of 4*4 image blocks are obtained.

[0131] In some optional embodiments, dividing the depth map formed by the super-resolution pixel depth features into blocks to obtain the super-resolution pixel depth features of each block includes: dividing the depth map formed by the super-resolution pixel depth features into blocks according to preset intervals to obtain the super-resolution pixel depth features within each preset size. The preset interval may be at least one of a length, a width, an area, or a number of pixels.

[0132] Step 508 , performing distribution statistics on the depth values ​​of the super-resolved pixel depth features of each block to obtain super-resolved local depth features.

[0133] The super-resolution local depth feature includes the depth value distribution information of the super-resolution pixel depth feature of each block. Since the depth value distribution statistics are adopted, the super-resolution local depth feature can represent the depth value change of each block to reflect the depth information at the local scale in detail. Optionally, in the case where part of the light and shadow of the super-resolution image data is only located in a certain image block of its depth map, the super-resolution local depth feature can reflect the depth information of this part of the light and shadow. Optionally, the super-resolution local depth feature includes but is not limited to: the variance, standard deviation, mean square error or correlation coefficient of the depth value.

[0134] Optionally, there are M blocks of super-resolution pixel depth features, and each block has N super-resolution pixel depth features. Then, based on the N super-resolution pixel depth features of each block, the super-resolution pixel depth features of the M blocks are determined, and the super-resolution pixel depth features of the M blocks are used as super-resolution local depth features.

[0135] In some optional embodiments, the depth values ​​of the super-resolution pixel depth features of each block are distributed and counted to obtain the super-resolution local depth features, including: performing variance calculation based on the depth values ​​of the super-resolution pixel depth features of each block to obtain the variance of the super-resolution pixel depth features of each block; using the variance of the super-resolution pixel depth features of each block as the super-resolution local depth features. Optionally, the size of the super-resolution pixel depth feature variance is positively correlated with the degree of change in the depth value of the corresponding area; if the variance is smaller, it means that the depth value of the area does not change much, that is, the depth is more uniform; if the variance is larger, it means that the depth value of the area changes more, and there may be obvious changes in texture or edges.

[0136] In this embodiment, the initial pixel depth features and the super-resolution pixel depth features are divided into blocks, and the distribution statistics of the depth values ​​of each pixel block can be performed. The local depth features can be determined from the neighborhood of the pixel itself to more richly reflect the local details of the depth features.

[0137] In some optional embodiments, the initial regional depth feature includes the initial global depth feature, the super-resolution regional depth feature includes the super-resolution global depth feature, and both the initial global depth feature and the super-resolution global depth feature are depth features of the global scale; that is, one of the regional scales is the global scale, the global scale includes each pixel, and the depth feature of the global scale is obtained based on the statistics of the depth features of each pixel; correspondingly, the first pixel point in the global scale corresponds to each pixel of the depth map constituted by the initial depth feature; the second pixel point in the global scale corresponds to each pixel of the depth map constituted by the super-resolution depth feature.

[0138] like Figure 6As shown, based on the feature distribution of the initial pixel depth feature, the initial region depth feature is determined, including step 602; correspondingly, based on the feature distribution of the super-resolution pixel depth feature, the super-resolution region depth feature is determined, including step 604, wherein:

[0139] Step 602 , counting the frequencies of the depth values ​​contained in the initial pixel depth feature to obtain an initial global depth feature.

[0140] The initial global depth feature represents the distribution of depth values ​​contained in the initial pixel depth feature, that is, the number of each depth value. Optionally, the initial global depth feature does not involve the pixel position in the initial pixel depth feature. Optionally, the initial global depth feature can be the frequency of the depth value itself, or can be the result of normalizing the frequency of the depth value.

[0141] In some optional embodiments, the frequencies of the depth values ​​contained in the initial pixel depth features are counted to obtain the initial global depth features, including: converting the depth values ​​contained in the initial pixel depth features into histogram data; and obtaining the initial global depth features based on the histogram data. Thus, the global depth features are directly determined through the histogram, so that the global depth features can be displayed in the form of an image, which is convenient for adjustment.

[0142] In some optional embodiments, the frequency of the depth value contained in the initial pixel depth feature is counted to obtain the initial global depth feature, including: taking the depth value contained in the initial pixel depth feature as a statistical dimension, taking the frequency as the value under the statistical dimension for statistics, and taking the statistical result as the initial global depth feature. Thus, directly determining the global depth feature through data can ensure processing efficiency.

[0143] Step 604 , counting the frequencies of the depth values ​​contained in the super-resolved pixel depth features to obtain super-resolved global depth features.

[0144] The super-resolution global depth feature represents the distribution of depth values ​​contained in the super-resolution pixel depth feature, that is, the number of each depth value. Optionally, the super-resolution global depth feature can be the frequency of the depth value itself, or can be the result of normalizing the frequency of the depth value.

[0145] In some optional embodiments, the frequency of the depth values ​​contained in the super-resolution pixel depth feature is counted to obtain the super-resolution global depth feature, including: converting the depth values ​​contained in the super-resolution pixel depth feature into histogram data; and obtaining the super-resolution global depth feature through the histogram data. Thus, the global depth feature is directly determined through the histogram, so that the global depth feature can be displayed in the form of an image, which is convenient for adjustment.

[0146] In some optional embodiments, the frequency of the depth value contained in the super-resolution pixel depth feature is counted to obtain the super-resolution global depth feature, including: taking the depth value contained in the super-resolution pixel depth feature as a statistical dimension, taking the frequency as a value under the statistical dimension, and counting the super-resolution pixel depth feature to obtain the super-resolution global depth feature. Thus, directly determining the global depth feature through data can ensure processing efficiency.

[0147] In this embodiment, statistics are performed based on the frequency of the depth values, and the number or proportion of each depth value can be counted to determine the overall depth information and depth features, which more fully reflects the overall depth information of the depth features.

[0148] In some optional embodiments, the initial depth features at multiple scales include initial pixel depth features and initial local depth features, the super-resolution depth features at multiple scales include super-resolution pixel depth features and super-resolution local depth features, and the depth loss at each scale includes pixel depth loss and local depth loss.

[0149] For each scale, the depth loss of each scale is determined based on the difference between the initial depth feature of the first pixel and the super-resolution depth feature of the second pixel, including: determining the pixel depth loss based on the feature value difference between the initial pixel depth feature of the first pixel and the super-resolution pixel depth feature of the second pixel; determining the local depth loss based on the feature value difference between the initial local depth feature of the first pixel and the super-resolution local depth feature of the second pixel.

[0150] The feature difference value is the difference in feature values ​​at the same scale. The feature difference value can reflect the similarity of different depth features at the same scale. Optionally, the feature difference value can be the difference in feature values, the absolute value of the difference, or the averaged result of the absolute value. For example, the feature difference value can be the L1 distance determined based on the L1 loss function and the initial pixel depth feature.

[0151] Since the initial pixel depth feature of the first pixel includes a depth value, and the super-resolution pixel depth feature of the second pixel also includes a depth value, a difference value can be directly calculated based on the two depth values ​​to obtain the pixel depth loss.

[0152] Since both the initial local depth feature and the super-resolution local depth feature are depth value distribution information, and the depth value distribution information includes but is not limited to the variance, standard deviation, mean square error or correlation coefficient of the depth value, the local depth loss is determined based on the difference between the depth value distribution information at the local scale.

[0153] For example: the depth value distribution information can be represented by variance, and the local depth loss can be represented by L1 distance; since the initial local depth feature of the first pixel point is a variance, and the super-resolution local depth feature of the second pixel point is another variance; and the L1 distance is calculated based on these two variances, and the L1 distance is used as the local depth loss.

[0154] Optionally, since there are multiple copies of pixels and blocks, the first pixel point and the second pixel point at the local scale and the pixel scale are multiple copies, and therefore it is necessary to perform feature fusion based on the difference in feature values ​​of each copy to obtain the depth loss of the corresponding scale.

[0155] In some optional embodiments, the pixel depth loss is determined based on the feature value difference between the initial pixel depth feature of the first pixel and the super-resolution pixel depth feature of the second pixel, including: performing difference calculation based on the absolute value of the depth value contained in the initial pixel depth feature of each first pixel and the absolute value of the depth value contained in the super-resolution pixel depth feature of each second pixel to obtain the difference in the absolute value of each pixel pair; performing averaging processing based on the difference in the absolute value of each pixel pair to obtain the pixel depth loss. Thus, the difference in absolute value is first determined, and then averaging processing is performed based on the difference to obtain the pixel depth loss, so as to avoid the extreme value of a certain pixel pair affecting the pixel depth loss.

[0156] In some optional embodiments, the pixel depth loss is determined based on the characteristic value difference between the initial pixel depth feature of the first pixel and the super-resolution pixel depth feature of the second pixel, including: calculating the L1 distance based on the characteristic value between the initial pixel depth feature of the first pixel and the super-resolution pixel depth feature of the second pixel, obtaining the L1 distance of each pixel pair, the L1 distance of each pixel pair is the characteristic value difference between the initial pixel depth feature of the first pixel and the super-resolution pixel depth feature of the second pixel; performing data fusion on the L1 distance of each pixel pair to obtain the pixel depth loss. The data fusion method includes but is not limited to averaging, filtering, screening, etc. based on the L1 distance of each pixel pair.

[0157] In some optional embodiments, the local depth loss is determined based on the feature value difference between the initial local depth feature of the first pixel and the super-resolution local depth feature of the second pixel, including: performing difference calculation based on the absolute value of the depth value contained in the initial local depth feature of each first pixel and the absolute value of the depth value contained in the super-resolution local depth feature of each second pixel to obtain the difference of the absolute value of each pixel block pair; performing averaging processing based on the difference of the absolute value of each pixel block pair to obtain the local depth loss. Thus, the difference of the absolute value is first determined, and then the averaging processing is performed based on the difference to obtain the local depth loss, so as to avoid the extreme value of a certain pixel pair affecting the local depth loss.

[0158] In some optional embodiments, the local depth loss is determined based on the feature value difference between the initial local depth feature of the first pixel and the super-resolution local depth feature of the second pixel, including: calculating the L1 distance based on the feature value difference between the initial local depth feature of the first pixel and the super-resolution local depth feature of the second pixel to obtain the L1 distance of each block of pixel pairs; the L1 distance of each block of pixel pairs is the feature value difference between the initial local depth feature of the first pixel and the super-resolution local depth feature of the second pixel; performing data fusion on the L1 distance of each block of pixel pairs to obtain the local depth loss. Among them, the data fusion method includes but is not limited to averaging, filtering, screening, etc. based on the L1 distance of each block of pixel pairs.

[0159] In this embodiment, the initial depth features and super-resolution depth features of the first pixel and the second pixel at the pixel scale and the local scale both have eigenvalues ​​related to the pixel position, so the eigenvalue differences are determined respectively to accurately determine the pixel depth loss and the local depth loss.

[0160] In some optional embodiments, the initial depth features of multiple scales include initial global depth features, the super-resolution depth features of multiple scales include super-resolution global depth features, and the depth loss of each scale includes global depth loss. For each scale, based on the difference between the initial depth feature of the first pixel and the super-resolution depth feature of the second pixel, the depth loss of each scale is determined, including: determining the depth offset of the super-resolution global depth feature of the second pixel relative to the initial global depth feature of the first pixel; converting the depth offset with the super-resolution global depth feature of the second pixel to obtain a global difference, and obtaining a global difference; based on the global difference, determining the global depth loss.

[0161] The depth offset is the depth difference with the initial global depth feature as the reference distribution, indicating the deviation of the initial global depth feature. Optionally, the depth offset can be the ratio between the initial depth feature of the first pixel and the super-resolution depth feature of the second pixel, or the normalized result of the ratio. Optionally, the depth offset can be the difference between the initial depth feature of the first pixel and the super-resolution depth feature of the second pixel, or the ratio between the difference and the preset reference data.

[0162] The global difference is asymmetric and is used to indicate the degree to which the super-resolved global depth approximates the initial global depth. Optionally, the global difference is the KL divergence.

[0163] In some optional embodiments, converting the depth offset and the super-resolution global depth feature of the second pixel to obtain a global difference includes: fusing a logarithm composed of a preset base of a real number of the depth offset with the super-resolution global depth feature of the second pixel to obtain a global difference. Wherein, when the preset base is 2, the logarithm represents the amount of information converted from the depth offset.

[0164] In some optional embodiments, the depth offset is converted with the super-resolution global depth feature of the second pixel to obtain the global difference, including: converting the depth offset into information; multiplying the information with the super-resolution global depth feature of the second pixel to obtain the global difference. Thus, KL divergence is used as the global difference. Since the process of converting to information introduces a logarithmic function, KL divergence can be made a nonlinear measurement method. The nonlinear characteristics of the logarithmic function help to amplify the impact of events with small probabilities and large differences.

[0165] In some optional embodiments, determining the global depth loss based on the global difference includes: using the global difference as the global depth loss, thereby improving processing efficiency.

[0166] In some optional embodiments, determining the global depth loss based on the global difference includes: adjusting the global difference according to the global coefficient to obtain the global depth loss, thereby ensuring the flexibility of the global depth loss.

[0167] In this embodiment, since the depth offset is a depth difference distributed with the super-resolved global depth feature as a reference, it can form an asymmetric global difference, which helps to more accurately reflect the degree to which the super-resolved global depth feature is similar to the initial global depth feature, thereby obtaining a more accurate global depth loss.

[0168] In some optional embodiments, the method further comprises: determining image data loss according to a pixel difference between the super-resolved image data and a reference image data, the resolution of the reference image data being greater than the resolution of the initial image data.

[0169] Correspondingly, the generator model to be trained is adjusted according to the depth loss, including: adjusting the generator model to be trained according to the depth loss and the image data loss.

[0170] The benchmark image data includes image data with a resolution higher than a resolution threshold. The benchmark image data is part of the sample data in the model training process. The benchmark image data includes at least the pixel value of each pixel, which can reflect the depth of the benchmark image data. Optionally, the benchmark image data is data belonging to the benchmark image (Ground Truth, GT) in the adversarial generative network. Optionally, the benchmark image data comes from a pre-acquired dataset, which can be acquired by a camera.

[0171] The pixel difference may be a difference between the pixel value of the reference image data and the pixel value of the super-resolution image data; the pixel difference may also be a difference between the absolute value of the pixel value of the reference image data and the absolute value of the pixel value of the super-resolution image data; the pixel difference may also be a ratio of the pixel value distribution between the super-resolution image data and the reference image data.

[0172] Image data loss, used to evaluate the pixel retention of the generator model to be trained. Optionally, the image data loss can be obtained by substituting pixel differences into the loss function for adjustment; in the case where the image data loss is pixel differences, it can be evaluated more efficiently; in the case where the image data loss is obtained by adjusting pixel differences based on the loss function, the image data loss can be determined more accurately. Optionally, the loss function can be a loss function of regularization terms such as L1 distance and L2 distance.

[0173] In some optional embodiments, the image data loss is determined based on the pixel difference between the super-resolution image data and the reference image data, including: calculating the L1 distance based on the difference between the pixel value of the first pixel and the pixel value of the reference image data of the third pixel to obtain the L1 distance between the first pixel and the third pixel; performing data fusion on the L1 distance between the first pixel and the third pixel to obtain the image data loss. The data fusion method includes but is not limited to performing averaging, filtering, screening, etc. based on the L1 distance between the first pixel and the third pixel. Thus, the image data loss is determined using the L1 loss function to accurately and efficiently quantify the pixel difference.

[0174] In some optional embodiments, the generator model to be trained is adjusted according to the depth loss and the image data loss, including: weighted fusion of the depth loss and the image data loss to obtain the fused loss; and adjusting the generator model to be trained by the fused loss. Thus, the depth loss and the image data loss are adaptively adjusted to more accurately control the speed and direction of adjusting the model weights of the generator model to be trained.

[0175] In some optional embodiments, the generator model to be trained is adjusted by the fused loss, including: performing gradient descent processing according to the fused loss to obtain the gradient of each parameter adjustment contained in the loss function; updating the model weight of the generator model to be trained according to the gradient until the stop training condition corresponding to the generator model to be trained is met. Thus, back propagation is performed by the gradient descent algorithm to more accurately control the speed and direction of the adjustment of the model weight of the generator model to be trained.

[0176] In some optional embodiments, the generator model to be trained is adjusted by the fusion loss, including: according to the fusion loss, from the output layer of the generator model to be trained to the input layer of the generator model to be trained, the gradient of each layer is calculated layer by layer; according to the gradient of each layer, the weight of each layer is updated to obtain the adjusted generator model, so as to obtain the target generator model according to the adjusted generator model that meets the preset conditions. Thus, by calculating the gradient algorithm of each layer layer by layer, the speed and direction of adjusting the model weight of the generator model to be trained can be more accurately controlled.

[0177] Optionally, the generator model to be trained is adjusted by the post-fusion loss, including: in the first stage, the first stage adjustment of the generator model to be trained is completed according to the post-fusion loss to obtain the adjusted generator model; in the second stage, the discriminator is adjusted to obtain the adjusted discriminator network; in the third stage, the target generator model is determined based on the adjusted generator model and the adjusted discriminator network after alternating training until the training stop condition is met.

[0178] In this embodiment, the depth loss and image data loss are combined to adjust the model of the generator to be trained, and the image super-resolution, enhancement and depth information retention capabilities of the generator to be trained can be adjusted synchronously to ensure the training speed.

[0179] In some optional embodiments, the method further includes: performing feature extraction on the super-resolution image data to obtain super-resolution image features; performing feature extraction on the reference image data to obtain reference image features; and determining image feature loss based on feature differences between the super-resolution image features and the reference image features.

[0180] Correspondingly, the generator model to be trained is adjusted according to the depth loss and the image data loss, including: adjusting the generator model to be trained according to the depth loss, the image data loss and the image feature loss.

[0181] The super-resolution image features are the features of the pixel values ​​contained in the super-resolution image data. The reference image features are the features of the pixel values ​​contained in the reference image data. The super-resolution image features and the reference image features are extracted by pre-trained neural networks respectively. Optionally, the super-resolution image features and the reference image features belong to the features extracted by a certain convolution layer of a neural network in a certain convolution block; optionally, the pre-trained neural network can be VGG-16 or VGG-19. Taking VGG-19 as an example, the super-resolution image features and the reference image features can be the feature maps output by convX_Y of VGG-19, that is, the features extracted by the Yth convolution layer in the Xth convolution block; for example, it includes conv3_4, that is, the feature map output by the 4th convolution layer in the 3rd convolution block.

[0182] The feature difference may be the difference between the feature value of the reference image feature and the feature value of the super-resolution image feature; the feature difference may also be the difference between the absolute value of the feature value of the reference image feature and the absolute value of the feature value of the super-resolution image feature; the feature difference may also be the ratio of the feature value distribution between the super-resolution image feature and the reference image feature.

[0183] Image feature loss, used to evaluate the pixel feature retention of the generator model to be trained. Optionally, the image feature loss can be obtained by substituting the feature difference into the loss function for adjustment; when the image feature loss is the feature difference, it can be evaluated more efficiently; when the image feature loss is obtained by adjusting the feature difference based on the loss function, the image feature loss can be determined more accurately. Optionally, the loss function can be a loss function of regularization terms such as L1 distance and L2 distance.

[0184] In some optional embodiments, the generator model to be trained is adjusted according to the depth loss, image data loss and image feature loss, including: weighted fusion of the depth loss, image data loss and image feature loss to obtain the fused loss; and adjusting the generator model to be trained by the fused loss. Thus, the depth loss, image data loss and image feature loss are synchronously incorporated into the training process to ensure processing efficiency.

[0185] In this embodiment, by combining depth loss, image data loss and image feature loss, the image super-resolution, enhancement and depth information retention capabilities of the generator to be trained can be adjusted synchronously, and due to the addition of image feature loss, the generator to be trained can output clearer image data without changing the resolution.

[0186] In some optional embodiments, Figure 7 As shown, according to the depth loss, the generator model to be trained is adjusted to obtain the target generator model, including:

[0187] Step 702: adjust the generator model to be trained according to the depth loss to obtain an adjusted generator model.

[0188] The adjusted generator model is a generator model obtained by adjusting the generator model to be trained at least once. Optionally, the adjusted generator model is a generative neural network model that can reconstruct image data with higher resolution based on the image data input therein.

[0189] In some optional embodiments, the generator model to be trained is adjusted according to the depth loss to obtain an adjusted generator model, including: weighted fusion of the depth loss and the image data loss to obtain a first fused loss; or weighted fusion of the depth loss, the image data loss and the image feature loss to obtain a second fused loss; according to the first fused loss or the second fused loss, the generator model to be trained is adjusted to obtain an adjusted generator model.

[0190] Step 704: super-resolution the image data to be discriminated by using the adjusted generator model to obtain super-resolution image data to be discriminated.

[0191] The image data to be judged is image data with a resolution lower than a resolution threshold. The image data to be judged belongs to part of the sample data in the model training process. The image data to be judged includes at least the pixel value of each pixel, and the pixel value can reflect the depth of the image data to be judged. Optionally, the image data to be judged is data belonging to a low-resolution image (LR) in a generative adversarial network. The image data to be judged and the initial image data may be the same or different image data.

[0192] The super-resolution image data to be identified is the result of super-resolution processing by the adjusted generator model. Since super-resolution processing can improve the resolution of the image data to be identified, the resolution of the super-resolution image data to be identified is higher than the resolution of the initial image data. The super-resolution image data to be identified and the super-resolution image data to be identified can be the same or different image data.

[0193] In some optional embodiments, based on the adjusted generator model, super-resolution processing is performed on the image data to be discriminated to obtain the super-resolution image data to be discriminated, including: based on the generator model with a U-shaped structure, super-resolution processing is performed on the image data to be discriminated to obtain the super-resolution image data to be discriminated. Among them, the processing method of the generator model with a U-shaped structure includes: inputting the image data to be discriminated into an encoder with gradually decreasing spatial resolution and gradually increasing feature dimensions to obtain a high-level semantic feature map; then passing through a decoder with gradually increasing spatial resolution and gradually decreasing feature dimensions, and finally passing through an upsampling module to obtain the super-resolution image data to be discriminated.

[0194] Step 706 , when the training stop condition is not met and the discriminator model to be trained detects that the super-resolved image data to be discriminated belongs to the reference image data, the discriminator model to be trained is adjusted to obtain an adjusted discriminator model.

[0195] The training stop condition is a condition for stopping the training of the discriminator model and the generator model to be trained. Optionally, the training stop condition can be the number of alternating training rounds of the generator model to be trained and the discriminator model to be trained, or the number of training rounds of the generator model to be trained.

[0196] The reference image data includes image data with a resolution higher than a resolution threshold. The reference image data is part of the sample data in the model training process. The reference image data includes at least the pixel value of each pixel, which can reflect the depth of the reference image data. Optionally, the reference image data is data belonging to the reference image (Ground Truth, GT) in the adversarial generative network, and the reference image data and the reference image data may be the same or different data. Optionally, the reference image data comes from a pre-acquired dataset, which may be acquired by a camera. The reference image data is not obtained by super-resolution of the adjusted generator model.

[0197] The discriminator model to be trained is used to determine whether the super-resolved image data to be judged belongs to the reference image data. Optionally, the discriminator model to be trained is used to output a confidence level, which represents whether the image is the reference image data.

[0198] Since the super-resolved image data to be discriminated is generated based on the adjusted generator model, when the discriminator model to be trained detects that the super-resolved image data to be discriminated belongs to the reference image data, the discriminator model to be trained makes an incorrect judgment, and thus the discriminator model to be trained needs to be adjusted.

[0199] In some optional embodiments, adjusting the discriminator model to be trained to obtain the adjusted discriminator model includes: adjusting the discriminator model to be trained until a preset number of rounds is reached, thereby obtaining the adjusted discriminator model.

[0200] In some optional embodiments, the discriminator model to be trained is adjusted to obtain an adjusted discriminator model, including: substituting the super-resolved image data to be discriminated and the reference image data into a binary cross entropy loss function to obtain a binary cross entropy loss value; and adjusting the discriminator model to be trained according to the binary cross entropy loss value to obtain the adjusted discriminator model.

[0201] Step 708, when the training stop condition is met and the adjusted discriminator model detects that the super-resolved image data to be discriminated belongs to the reference image data, a target generator model is obtained according to the adjusted generator model.

[0202] In some optional embodiments, obtaining the target generator model according to the adjusted generator model includes: using the adjusted generator model as the target generator model. Thus, directly using the adjusted generator model as the target generator model has a relatively high processing efficiency.

[0203] In some optional embodiments, obtaining a target generator model according to the adjusted generator model includes: substituting the adjusted generator model into subsequent model training, and obtaining the target generator model according to the result of the subsequent model training. Thus, the adjusted generator model is substituted into the training process of other subsequent models as a basis, so as to adjust the required target generator model for certain tasks.

[0204] Since the super-resolution image data to be judged is generated based on the adjusted generator model, and the adjusted discriminator model detects that the super-resolution image data to be judged belongs to the reference image data, it means that the adjusted generator model has reached the corresponding standard. Therefore, there is no need to optimize according to the adjusted discriminator model at the current moment, but the training process of the adjusted generator model is controlled based on the training stop condition.

[0205] In this embodiment, the adjusted generator model and the adjusted discriminator model are first obtained respectively, rather than directly conducting adversarial training, to avoid the unstable convergence problem in direct adversarial training. After the separate training of the generator model to be trained is completed, the super-resolved image data to be discriminated is obtained, and the discriminator model to be trained is trained with the super-resolved image data to be discriminated and the reference image data, and then when the adjusted discriminator model detects that the super-resolved image data to be discriminated belongs to the reference image data, the target generator model is determined based on the adjusted generator model to form a generative adversarial network, thereby ensuring that the image data output by the target generator model retains specific depth information.

[0206] In some optional embodiments, before the training stop condition is met, the method also includes: when the discriminator model or the adjusted discriminator model detects that the image data to be detected does not belong to the reference image data, the adjusted generator model is used as the generator model to be trained, and the above-mentioned step of super-resolution processing of the initial image data based on the generator model to be trained is returned to execute to obtain super-resolution image data.

[0207] As for the step of using the adjusted generator model as the generator model to be trained, it can be achieved by adjusting the generator model to be trained according to the depth loss in step 702 to obtain the adjusted generator model. At this time, there is no need to retain the original generator model to be trained, and the adjusted generator model can directly replace the original generator model to be trained.

[0208] For the step of using the adjusted generator model as the generator model to be trained, there may be a step of data determination, that is, after obtaining the adjusted generator model, the original generator model to be trained and the adjusted generator model are retained at the same time.

[0209] Returning to the step of performing super-resolution processing on the initial image data based on the generator model to be trained to obtain super-resolution image data means re-executing steps 202 and 204 to implement the update of the adjusted generator model.

[0210] In this embodiment, when the discriminator model or the adjusted discriminator model makes correct judgments, it is necessary to return to the step of obtaining the super-resolution image data so that the adjusted generator model can be updated, thereby realizing the alternating training of the generator model to be trained and the discriminator model to be trained, thereby ensuring that the target generator model can more accurately retain the depth information of the image data to be adjusted.

[0211] In some optional embodiments, the training stop condition includes the number of alternating rounds reaching the number of training stop rounds; the number of alternating rounds is the number of rounds of alternating training of the generator model and the discriminator model to be trained; the number of training stop rounds is greater than the number of adjusted rounds of the adjusted discriminator model, and the number of training stop rounds is less than the number of adjusted rounds of the adjusted generator model.

[0212] The number of alternating rounds is obtained by counting the number of alternations; one round of alternating means that one round of training of the generator model to be trained and one round of training of the discriminator model to be trained are recorded as alternating training, one round of training of the generator model to be trained includes at least one batch of training of the generator model to be trained, and one round of training of the discriminator model to be trained includes at least one batch of training of the discriminator model to be trained; optionally, each round of training uses one batch of data. Exemplarily, there are 20,000 samples in the training set, a batch size is 100 samples, an epoch includes the training of 200 batches of samples, and each batch of samples is one round of training. Exemplarily, the number of training stop rounds can be 10 times.

[0213] Optionally, the step size used is 1, that is, the number of steps for each alternating training is 1. Optionally, training the generator model to be trained once means that the adjusted generator model is used as the generator model to be trained, and the samples of 1 epoch are used to adjust the generator model to be trained for one round; the discriminator model to be trained is adjusted for one round, which means that the adjusted discriminator model is used as the discriminator model to be trained, and the samples of 1 epoch are used to adjust the discriminator model to be trained for one round.

[0214] Optionally, the generator model to be trained is 1 epoch, that is, the forward propagation and backward propagation processes are completed once on the entire training set of the generator model to be trained, that is, one epoch is a complete traversal of the entire training data set of the generator model to be trained; the entire training set of the generator model to be trained includes at least initial image data, and may also include baseline image data and reference image data.

[0215] Optionally, the discriminator model to be trained is 1 epoch, that is, the forward propagation and backward propagation processes are completed once on the entire training set of the discriminator model to be trained, that is, one epoch is a complete traversal of the entire training data set of the discriminator model to be trained; the entire training set of the discriminator model to be trained includes at least the super-resolution image data to be discriminated, the benchmark image data, and the reference image data.

[0216] The number of rounds of adjustment of the adjusted generator model refers to the number of rounds of adjustment of the model parameters of the generator model to be trained during the separate training process of the generator to be trained until the convergence condition of the generator model is met. Optionally, the number of rounds of adjustment of the adjusted generator model is approximately greater than or equal to the first generator round threshold, and less than or equal to the second generator round threshold; the first generator round threshold is approximately 20, and the second generator round threshold is approximately 40.

[0217] The number of rounds of adjustment of the adjusted discriminator model is the number of rounds of adjustment of the model parameters of the discriminator model to be trained during the separate training process of the discriminator to be trained until the convergence condition of the discriminator model is met. Optionally, the number of rounds of adjustment of the adjusted discriminator model is approximately greater than or equal to the first discriminator round threshold, and less than or equal to the second discriminator round threshold; the second discriminator round threshold is approximately 5 or 6.

[0218] Exemplarily, the number of rounds of adjustment of the adjusted generator model is about 20 to 40 times, the number of rounds of adjustment of the adjusted discriminator model is about 5 times, and the number of training stop rounds is about 10 times.

[0219] In this embodiment, the ability of the generator to generate natural textures is enhanced through alternating training. At the same time, the number of training stop rounds is greater than the number of adjustment rounds of the adjusted discriminator model, and the number of training stop rounds is less than the number of adjustment rounds of the adjusted generator model, so as to show the number of rounds between the first stage of the adjusted discriminator model, the second stage of training the adjusted generator model alone, and the third stage of alternating training.

[0220] In one embodiment, an image processing method is provided. This embodiment is illustrated by applying the method to an electronic device, wherein the electronic device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc.

[0221] In this embodiment, Figure 8 As shown, the image processing method includes the following steps 802 to 804, wherein:

[0222] Step 802: Obtain image data to be adjusted.

[0223] The image data to be adjusted is image data that needs to be super-resolution processed. The image data to be adjusted does not belong to sample data.

[0224] In some optional embodiments, acquiring the image data to be adjusted includes: acquiring image data acquired by the electronic device through an image sensor, and using the image data acquired by the image sensor as the image data to be adjusted.

[0225] In some optional embodiments, obtaining the image data to be adjusted includes: obtaining image data acquired by an electronic device through an image sensor; performing image signal processing (ISP) on the image data acquired by the image sensor, and using a certain image processing result in the image signal processing process as the image data to be adjusted.

[0226] Step 804, super-resolution is performed on the image data to be adjusted through the target generator model to obtain target image data; wherein the target generator model is a generator model trained by the training embodiment of the above-mentioned image processing model.

[0227] The target image data is the result of super-resolution processing by the target generator model. The resolution of the target image data is higher than the resolution of the image data to be adjusted.

[0228] In some optional embodiments, based on the target generator model, super-resolution processing is performed on the image data to be adjusted to obtain target image data, including: upsampling the image data to be adjusted to obtain first image data to be fused; dividing the first image data to be fused into multiple blocks of image data to be fused, and obtaining the characteristic value of each block of image data to be fused; determining the filter corresponding to each block of image data to be fused according to the characteristic value of each block of image data to be fused, and using each filter to filter the corresponding image data block to be fused in the first image data to be fused to obtain second image data to be fused; fusing the second image data to be fused with the first image data to be fused to obtain target image data.

[0229] In some optional embodiments, based on the target generator model, super-resolution processing is performed on the image data to be adjusted to obtain the target image data, including: based on the target generator model with a U-shaped structure, super-resolution processing is performed on the image data to be adjusted to obtain the target image data. Among them, the processing method of the target generator model with a U-shaped structure includes: inputting the image data to be adjusted into an encoder with gradually decreasing spatial resolution and gradually increasing feature dimensions to obtain a high-level semantic feature map; then passing through a decoder with gradually increasing spatial resolution and gradually decreasing feature dimensions, and finally passing through an upsampling module to obtain the target image data.

[0230] In the above-mentioned image processing method, since the target generator model is trained based on depth difference, it can retain depth information more accurately to produce a more natural light and shadow transition and a better depth of field effect, so that the generated image is closer to the original image or real scene in terms of light and shadow perception, thereby enhancing the realism of the output image, solving the problem of spatial and stereoscopic loss in traditional generative neural networks in image super-resolution tasks, and improving the visual quality and aesthetic perception of the generated image. As a result, the original spatial and stereoscopic sense is effectively prevented from being lost during the image super-resolution process, and more image details are retained. In addition, the user's visual experience is significantly improved, making the image more beautiful and ornamental.

[0231] Deep learning neural networks can perform image processing techniques such as image super-resolution and image enhancement. Compared with ordinary non-generative neural networks, the generative deep learning networks represented by the above-mentioned generative adversarial networks have some outstanding advantages, including: (1) they can generate richer details and more complex textures, improving the visual quality of high-resolution images. In contrast, non-generative neural networks may produce smoother results; (2) through the adversarial learning mechanism between the generator and the discriminator, they can capture better edges and details, while suppressing artifacts and blur, and output high-resolution images that are closer to reality.

[0232] However, in the field of photography, especially mobile photography, generative adversarial networks need to be deployed on the end. Due to limitations such as the time consumption of model inference and the number of parameters, the model's learning ability is limited, which will bring some unavoidable problems: (1) Generative adversarial networks can capture high-level features of images, but when processing images with complex light and shadow and depth of field features, it may be difficult to restore natural details; (2) The powerful deblurring ability of generative adversarial networks may destroy the original natural edge transition and depth of field of the image, making high-resolution images have a "comic sense"; (3) During the training process of the generative adversarial network, the generator may pay too much attention to the objective visual quality and remove some actual noise during the imaging process. However, this over-optimization may lead to the loss of the original sense of space and three-dimensionality.

[0233] To address these issues, we introduce image depth estimation technology to guide and constrain the training of generative adversarial networks. This can improve the light and shadow fidelity and depth of field effects of the output image without increasing the model size (i.e., it can be deployed on the end), and reduce the loss of spatial and stereoscopic sense. Fig. 9 As shown in the figure, due to the changes in the edge transition and light and shadow of the original image by the generative adversarial image super-resolution network, the image depth has changed from left to right. Fig. 9 In (a), the foreground wall becomes farther away. Fig. 9 In (b), the depth of the leaves in the background is reduced. Fig. 9 In (c), the relief has a reduced three-dimensional effect. Fig. 9 The light and shadow sense in the picture (c) is reduced. Fig. 9 In (d), the layering of the flowers is reduced.

[0234] In one embodiment, Fig.10 As shown in Figure (a), the training phase uses a generative adversarial network for training, which includes the adjustment of the generator to be trained and the discriminator to be trained. The generator to be trained attempts to produce more and more realistic images to deceive the discriminator to be trained, while the discriminator to be trained improves its ability to distinguish whether the super-resolved image data is the reference image data. Through the two models that are in confrontation with each other, the two models are continuously adjusted and optimized to finally achieve the training goal; the training goal is to generate super-resolved image data close to the reference image data through the target generator model. Among them, the addition of the depth estimation network will perform depth estimation on the initial image data and the super-resolved image data respectively, and estimate the pixel depth loss based on the pixel-by-pixel distance, the block depth estimation variance as the local depth loss, and the histogram KL divergence as the global depth loss, increase the supervision of the generator model to be trained, constrain the training of the generator model to be trained, so that the depth of the super-resolved image data output by the adjusted generator model gradually approaches the depth of the initial image data until the target generator model is obtained.

[0235] Correspondingly, such as Fig.10 As shown in Figure (b), after obtaining the target generator model, the image data to be adjusted is super-resolved through the trained target generator model to obtain the target image data. Among them, the reference image data and the benchmark image data belong to the actual collected data (ground truth, GT) in the sample data, and the initial image data and the image data to be adjusted belong to the low-resolution image data (Low Resolution, LR).

[0236] In one embodiment, Fig.11 As shown, the initial image data and the super-resolution image data are input into the depth estimation network, and the depth estimation network performs feature extraction and upsampling on the initial image data to obtain the depth features of the initial image; at the same time, the depth estimation network performs feature extraction on the super-resolution image data to obtain the depth features of the super-resolution image; then, the pixel depth loss, the block depth estimation variance are estimated as the local depth loss, and the histogram KL divergence is used as the global depth loss, and then the pixel depth loss, the local depth loss and the global depth loss are added to obtain the depth loss.

[0237] Optionally, the generator to be trained and the discriminator to be trained in the training phase both belong to U-type networks; the generator to be trained is a U-type network that can be deployed on the end side. The input initial image data first passes through an encoder with gradually decreasing spatial resolution and gradually increasing feature dimensions to obtain a high-level semantic feature map. Then it passes through a decoder with gradually increasing spatial resolution and gradually decreasing feature dimensions. Finally, it passes through an upsampling module to obtain super-resolution image data. The discriminator to be trained is another U-type network with a smaller size than the generator to be trained. It is similar to the generator to be trained, but there is no upsampling module. It inputs SR or GT and outputs a tensor with the same spatial resolution as the input and the number of channels as 1. This tensor represents the confidence of the discriminator to be trained in judging that the input image is True in the pixel dimension.

[0238] Optionally, the depth estimation in the training phase uses the open source network Depth Anything V2 for image depth estimation, which is a basic model for monocular depth estimation with SOTA accuracy. Compared with the model based on stable diffusion (SD), Depth Anything V2 has faster inference speed, fewer parameters and higher depth accuracy, and is more suitable as a supervisory network for GAN training. The initial image data and super-resolution image data are input respectively, and the depth map with the same spatial resolution as the input and the number of channels as 1 will be obtained for the calculation of the subsequent loss function.

[0239] In one embodiment, the training phase is divided into three phases: training the generator alone, training the discriminator alone, and training the generator and the discriminator alternately. The supervision of depth estimation is involved in the first and third phases.

[0240] In the first stage, the weights of the discriminator to be trained and the depth estimation network are fixed, the initial image data is input into the generator to be trained, and the super-resolution image data is output; the initial image data and the super-resolution image data are respectively input into the depth estimation network, and the depth map composed of the initial image feature dep and the super-resolution image feature dep is output. The loss function of this stage includes the generator loss L G , depth loss L dep The generator loss L G The role of this is to enable the generator to have basic image super-resolution and enhancement capabilities.

[0241] Among them, L G The pixel-by-pixel L1 distance loss L1 and the perceptual loss L percep The distance loss L1 is determined by substituting the L1 loss function pixel by pixel. The distance loss L1 is the image data loss. The perceptual loss L is determined by substituting the pixel features into the L1 loss function. percep , the perceptual loss L percep is the image feature loss, which is expressed as follows:

[0242] L G =λL1+ηL percep

[0243]

[0244] Among them, λ and η are the weighted sums of each loss respectively. lr ,I gt Respectively represent the initial image data and the reference image data image, W and H represent the width and height of the output image, G θ represents the generator network to be trained with weight parameter θ, φ represents the pre-trained VGG-19 feature extraction network, φ i represents the feature map tensor extracted by the i-th convolutional layer of φ, where i is at least part of the convolutional layer selected by VGG-19. That is, the absolute pixel-by-pixel distance between the super-resolved image data and the reference image data is measured by the L1 loss function as the image data loss; L percep The absolute distance between the low-level and high-level semantic features extracted by the feature extraction network for the super-resolved image data and the reference image data is measured as the image feature loss. The addition of image feature loss will improve the clarity of the super-resolved image data. Optionally, the L1 loss function of each layer of pixel feature output is calculated as the weighted sum according to their respective weights, which is the perceptual loss.

[0245] At the same time, the depth loss Ldep This helps to make the image depth features of the super-resolved image data close to the original image data, even if the generator to be trained retains the depth features of the original image data. dep By the depth loss based on L1 loss function Deep loss L based on L1 loss function and block variance dep-Var , Deep loss L based on histogram KL divergence dep-Hist constitute.

[0246]

[0247] Among them, DE represents the depth estimation network, and up represents the interpolation upsampling process.

[0248] Measures the L1 absolute distance between the depth map composed of super-resolved depth features and the depth map composed of upsampled initial depth features.

[0249] L dep-Var Measures the difference in block variance between the depth map composed of super-resolved depth features and the depth map composed of upsampled initial depth features. The block variance between depth maps is the local depth difference. The two depth maps are divided into several image blocks of equal size, and the variance of the depth estimation value in each image block is counted to obtain the block variance map of each image. Substitute these two block variance maps into the L1 loss function to obtain the difference in block variance. For example: using 64*64 blocks, an image has a total of 4*4 image blocks, and the variance of the depth features of each 4*4 image block is counted to obtain the block variance map of each of the two images with a size of 4*4 to obtain the local depth loss.

[0250] L dep-Hist It measures the difference between the super-resolved depth features and the histogram of the initial depth features dep. The difference between the histograms is the global depth difference. hist represents the normalized depth histogram. j represents the jth item in the normalized histogram, P represents the softmax value (i.e., probability) of the jth item in the normalized histogram of the super-resolved depth feature dep, and Q represents the softmax value of the jth item in the normalized histogram of the initial depth feature dep. KL divergence is used to measure the distance between two distributions. Therefore, the histogram is converted into a distribution through softmax calculation. By minimizing KL divergence, the super-resolved depth feature can be brought closer to the initial depth feature.

[0251] These three depth loss items will make the high-resolution image output by the network closer to the input original image in depth (i.e., spatial stereoscopic sense) from different angles; among them, the pixel depth loss and the local depth loss are both calculated by the L1 loss function. The pixel depth loss is used to make the pixel values ​​similar, and the local depth loss is used to make the local variances in the neighborhood similar. The combined use has stronger constraints. The global depth loss is the KL divergence.

[0252] In the first stage above, minimizing L G +L dep After training for about 20 to 40 epochs, a basically useful generator to be trained will be obtained. At this time, the super-resolution image data generated by the generator to be trained has basic super-resolution, deblurring, and enhancement effects. At the same time, due to the constraint of depth loss, the light and shadow details of the initial image data are preserved. However, the super-resolution image data at this time lacks high-quality detail textures, and the visual experience is still not ideal.

[0253] In the second stage, the weights of the adjusted generator model and the depth estimation network are fixed, and the image data to be discriminated is input into the adjusted generator model to obtain the super-resolved image data to be discriminated. Then, the super-resolved image data to be discriminated and the reference image data are respectively input into the discriminator model to be trained, and the pixel-by-pixel confidence representing whether the super-resolved image data to be discriminated is True is output. The loss function of this stage is the model loss L of the discriminator to be trained D , L D It measures the ability of the discriminator model to be trained to distinguish whether the image belongs to the target distribution (i.e. True / False), which is calculated by the loss L of the discriminator model to be trained on the reference image data. D-gt , the loss L of the discriminator model to be trained for the super-resolved image data to be discriminated D-sr The composition is expressed as follows:

[0254] L D =L D-gt +L D-sr

[0255]

[0256] Where D represents the discriminator model to be trained, σ represents the Sigmoid normalization process, and BCELoss is the binary cross entropy loss, which measures the difference between two distributions in a binary classification problem. y is a training label tensor with a spatial resolution consistent with the input of the discriminator model to be trained, and the number of channels is 1. label＝1 That is, all elements of the tensor are 1, indicating that the label is True, y label＝0 That is, all elements of the tensor are 0, indicating that the label is False. D-gtFor example, the output of the discriminator model to be trained is the pixel-by-pixel confidence that the super-resolution image data to be discriminated is True. The spatial resolution of this tensor is consistent with the reference image data and the number of channels is 1. By minimizing L D-gt , will enable the discriminator model to be trained to more accurately determine whether the super-resolved image data to be discriminated is the reference image data. Similarly, L D-sr The output of the discriminator model to be trained is the pixel-by-pixel confidence that the super-resolved image data to be discriminated is False, which is obtained by minimizing L D-sr , which will enable the discriminator model to be trained to more accurately judge the super-resolved image data to be discriminated as a false image.

[0257] In the second stage above, minimizing L D-gt +L D-sr The number of epochs in training is much smaller than that in the first stage. For example, 5 epochs will obtain images that can basically correctly distinguish between real images and fake images generated by the adjusted generator model, preparing for the third stage of adversarial training.

[0258] In the third stage, after the previous two stages, a reliable generator model and a discriminator model to be trained have been obtained. It is necessary to enter the third stage to train the two networks adversarially to improve the ability of the generator model to be trained to generate natural textures. The weights of the depth estimation network are fixed, and the generator model to be trained and the discriminator model to be trained are trained alternately, that is, the weights of the discriminator model to be trained are fixed first to train the generator model to be trained, and then the weights of the generator model to be trained are fixed to train the discriminator model to be trained.

[0259] The step size used in this embodiment is 1, that is, the number of steps in each alternating training is 1, and the loss function used is consistent with the loss corresponding to the first and second stages, which is α G L G +α dep L dep +α D L D , where α G , α dep , α Dis the weight of each loss (this embodiment uses 1, 0.1, and 0.1). After alternating training, the optimization of the generator model to be trained and the discriminator model to be trained will produce confrontation. The generator model to be trained generates SR that is closer to GT, and the discriminator model to be trained tends to judge SR as a false image. Therefore, the generator model to be trained will tend to generate more realistic images, including natural noise and texture features in GT, thereby improving the image quality of SR. The depth estimation loss is added again at this stage, which can prevent the generator model to be trained from losing the natural light and shadow and depth of field features in the input due to excessive pursuit of being close to GT, improve the fidelity of the output image of the generator model to be trained, and retain the sense of space and three-dimensionality. In this stage, the number of epochs is set between the first stage and the second stage (the number of epochs used in this embodiment is 10).

[0260] In this embodiment, in the image super-resolution and image enhancement tasks of mobile phone photography, depth estimation is combined with the training of generative neural networks, and depth estimation loss is introduced to retain the spatial features and natural details of the image, improve the fidelity of the super-resolution image data, and improve the user's visual experience.

[0261] Among them, introducing the depth estimation network as supervision means: by introducing the latest monocular depth estimation model DepthAnything V2, the depth estimation loss is used to train the generator, so that the high-resolution image generated by the network not only retains the details, but also maintains the depth information of the input low-resolution image.

[0262] Multiple depth estimation losses refer to: depth loss determined based on the L1 loss function, depth loss based on block variance determined based on the L1 loss function, and depth estimation loss based on histogram KL divergence. They ensure the consistency of depth features and spatial sense of the generated image from multiple angles, and avoid distortion or loss of natural light and shadow effects to a greater extent.

[0263] The multi-stage alternating training strategy means that the image super-resolution capability of the generator is gradually improved by training the generator and the discriminator in stages, avoiding the unstable convergence problem in direct adversarial training. In the final stage, alternating training is used to enhance the generator's ability to generate natural textures.

[0264] Finally, in the inference stage, the target generator model trained in the training stage has high super-resolution and enhancement capabilities and subjective image quality perception and fidelity. In the inference stage, only the generator is retained and all weights and parameters are fixed. When the image data to be adjusted is input into the target generator model, high-resolution / enhanced target image data can be obtained.

[0265] In this embodiment, by combining the depth estimation network with the image super-resolution / enhancement generative adversarial network, the depth loss function is innovatively introduced to constrain network training, and the output image is made closer to the input image in terms of fidelity, light and shadow brightness, and depth of field through multi-stage training of the generator and the discriminator, thereby improving the realism and fidelity of the super-resolution image data. As a result, this embodiment can produce a more natural light and shadow transition and a better depth of field effect, so that the generated image is closer to the original image or the real scene in terms of light and shadow perception, thereby enhancing the realism of the output image, solving the problem of spatial sense and stereoscopic sense loss in the existing generative neural network in the image super-resolution / enhancement task, and improving the visual quality and aesthetic perception of the generated image to improve the image quality and fidelity. At the same time, it effectively prevents the loss of the original space and stereoscopic sense in the image super-resolution process, retains more image details, significantly improves the user's visual experience, makes the image more beautiful and ornamental, and better restores the picture details.

[0266] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0267] Based on the same inventive concept, the embodiment of the present application also provides a training device for an image processing model for implementing the training method of the image processing model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more image processing models provided below can refer to the limitations of the training method for the image processing model above, and will not be repeated here.

[0268] In an exemplary embodiment, Fig.12 As shown, a training device for an image processing model is provided, comprising:

[0269] The super-resolution module 1202 is used to perform super-resolution processing on the initial image data based on the generator model to be trained to obtain super-resolution image data; the resolution of the super-resolution image data is greater than the resolution of the initial image data;

[0270] A loss module 1204, configured to determine a depth loss of the generator model to be trained according to a depth difference between the super-resolved image data and the initial image data;

[0271] The training module 1206 is used to adjust the generator model to be trained according to the depth loss to obtain a target generator model.

[0272] In one embodiment, the loss module 1204 is used to:

[0273] Performing depth feature detection at multiple scales on the initial image data to obtain initial depth features at multiple scales;

[0274] Performing depth feature detection at multiple scales on the super-resolution image data to obtain super-resolution depth features at multiple scales;

[0275] For each of the scales, based on a difference between the initial depth feature of the first pixel and the super-resolved depth feature of the second pixel, determining the depth loss of each of the scales; matching a first pixel in the initial image data of the same scale with a second pixel in the super-resolved image data;

[0276] According to the depth losses of each of the scales, the depth loss of the generator model to be trained is determined.

[0277] In one of the embodiments, the initial depth features of the multiple scales include initial pixel depth features and initial region depth features, and the super-resolved depth features of the multiple scales include super-resolved pixel depth features and super-resolved region depth features;

[0278] The loss module 1204 is used to:

[0279] Performing depth detection on the initial image data to obtain the initial pixel depth feature;

[0280] Determining the initial region depth feature based on the feature distribution of the initial pixel depth feature;

[0281] Performing depth detection on the super-resolution image data to obtain the super-resolution pixel depth feature;

[0282] The super-resolved region depth feature is determined based on the feature distribution of the super-resolved pixel depth feature.

[0283] In one of the embodiments, the initial pixel depth feature and the super-resolution pixel depth feature are detected by a depth estimation network; the weight of the depth estimation network is fixed, or the weight of the depth estimation network is updateable.

[0284] In one of the embodiments, the initial region depth feature includes an initial local depth feature, and the super-resolved region depth feature includes a super-resolved local depth feature;

[0285] The loss module 1204 is used to:

[0286] Dividing the depth map formed by the initial pixel depth features into blocks to obtain initial pixel depth features of each block;

[0287] Performing distribution statistics on the depth values ​​of the initial pixel depth features of the blocks to obtain the initial local depth features;

[0288] Dividing the depth map formed by the super-resolution pixel depth features into blocks to obtain super-resolution pixel depth features of each block;

[0289] The depth values ​​of the super-resolution pixel depth features of each block are distributed and counted to obtain the super-resolution local depth features.

[0290] In one of the embodiments, the initial region depth feature includes an initial global depth feature, and the super-resolved region depth feature includes a super-resolved global depth feature;

[0291] The loss module 1204 is used to:

[0292] Counting the frequencies of the depth values ​​contained in the initial pixel depth feature to obtain the initial global depth feature;

[0293] The frequencies of the depth values ​​contained in the super-resolution pixel depth feature are counted to obtain the super-resolution global depth feature.

[0294] In one embodiment, the loss module 1204 is used to:

[0295] Performing depth detection on the initial image data to obtain depth features to be sampled of the initial image data;

[0296] According to the size of the super-resolution image data, the depth feature to be sampled is up-sampled to obtain the initial pixel depth feature.

[0297] In one of the embodiments, the initial depth features of the multiple scales include initial pixel depth features and initial local depth features, the super-resolved depth features of the multiple scales include super-resolved pixel depth features and super-resolved local depth features, and the depth loss of each scale includes pixel depth loss and local depth loss;

[0298] The loss module 1204 is used to:

[0299] Determining the pixel depth loss based on a feature value difference between the initial pixel depth feature of the first pixel and the super-resolution pixel depth feature of the second pixel;

[0300] The local depth loss is determined based on a feature value difference between the initial local depth feature of the first pixel and the super-resolved local depth feature of the second pixel.

[0301] In one embodiment, the initial depth features of the multiple scales include initial global depth features, the super-resolved depth features of the multiple scales include super-resolved global depth features, and the depth loss of each scale includes a global depth loss;

[0302] The loss module 1204 is used to:

[0303] Determining a depth offset of the super-resolved global depth feature of the second pixel relative to the initial global depth feature of the first pixel;

[0304] Converting the depth offset and the super-resolved global depth feature of the second pixel to obtain a global difference;

[0305] Based on the global disparity, the global depth loss is determined.

[0306] In one embodiment, the loss module 1204 is used to:

[0307] determining image data loss based on a pixel difference between the super-resolved image data and a reference image data, the resolution of the reference image data being greater than the resolution of the initial image data;

[0308] Correspondingly, the training module 1206 is used to:

[0309] The generator model to be trained is adjusted according to the depth loss and the image data loss.

[0310] In one embodiment, the loss module 1204 is used to:

[0311] Performing feature extraction on the super-resolution image data to obtain super-resolution image features;

[0312] Performing feature extraction on the reference image data to obtain reference image features;

[0313] Determining image feature loss according to a feature difference between the super-resolved image feature and the reference image feature;

[0314] Correspondingly, the training module 1206 is used to:

[0315] The generator model to be trained is adjusted according to the depth loss, image data loss and image feature loss.

[0316] In one embodiment, the training module 1206 is used to:

[0317] According to the depth loss, adjusting the generator model to be trained to obtain an adjusted generator model;

[0318] The adjusted generator model is used to perform super-resolution processing on the image data to be discriminated, so as to obtain super-resolution image data to be discriminated;

[0319] When the training stop condition is not met and the discriminator model to be trained detects that the super-resolved image data to be discriminated belongs to the reference image data, adjusting the discriminator model to be trained to obtain an adjusted discriminator model;

[0320] When the training stop condition is met and the adjusted discriminator model detects that the super-resolved image data to be discriminated belongs to the reference image data, a target generator model is obtained according to the adjusted generator model.

[0321] In one embodiment, before the training stop condition is met, the training module 1206 is used to:

[0322] When the discriminator model to be trained or the adjusted discriminator model detects that the image data to be detected does not belong to the reference image data, the adjusted generator model is used as the generator model to be trained, and the step of performing super-resolution processing on the initial image data based on the generator model to be trained is returned to execute to obtain super-resolution image data.

[0323] In one embodiment, the training stop condition includes the number of alternating rounds reaching the number of training stop rounds; the number of alternating rounds is the number of rounds of alternating training of the generator model to be trained and the discriminator model; the number of training stop rounds is greater than the number of rounds adjusted by the adjusted discriminator model, and the number of training stop rounds is less than the number of rounds adjusted by the adjusted generator model.

[0324] Each module in the training device of the above-mentioned image processing model can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0325] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0326] In an exemplary embodiment, Fig.13 As shown, an image processing device is provided, comprising:

[0327] An acquisition module 1302 is used to acquire image data to be adjusted;

[0328] The super-resolution module 1304 is used to perform super-resolution processing on the image data to be adjusted through a target generator model to obtain target image data; wherein the target generator model is a generator model trained by the training device of the above-mentioned image processing model.

[0329] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as follows: Fig.14 As shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for an image processing model is implemented.

[0330] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.15As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0331] Those skilled in the art will understand that Fig.15 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0332] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0333] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0334] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0335] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0336] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0337] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present application. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several deformations and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the attached claims.

Claims

1. A training method for an image processing model, characterized in that: The method comprises: Based on the generator model to be trained, super-resolution processing is performed on the initial image data to obtain super-resolution image data; the resolution of the super-resolution image data is greater than the resolution of the initial image data; Determining a depth loss of the generator model to be trained according to a depth difference between the super-resolved image data and the initial image data; According to the depth loss, the generator model to be trained is adjusted to obtain a target generator model.

2. The method according to claim 1, characterized in that The step of determining the depth loss of the generator model to be trained according to the depth difference between the super-resolved image data and the initial image data comprises: Performing depth feature detection at multiple scales on the initial image data to obtain initial depth features at multiple scales; Performing depth feature detection at multiple scales on the super-resolution image data to obtain super-resolution depth features at multiple scales; For each of the scales, based on a difference between the initial depth feature of the first pixel and the super-resolved depth feature of the second pixel, determining the depth loss of each of the scales; matching a first pixel in the initial image data of the same scale with a second pixel in the super-resolved image data; According to the depth losses of each of the scales, the depth loss of the generator model to be trained is determined.

3. The method according to claim 2, characterized in that The initial depth features of the multiple scales include initial pixel depth features and initial region depth features, and the super-resolution depth features of the multiple scales include super-resolution pixel depth features and super-resolution region depth features; The performing depth feature detection on the initial image data at multiple scales to obtain initial depth features at multiple scales includes: Performing depth detection on the initial image data to obtain the initial pixel depth feature; Determining the initial region depth feature based on the feature distribution of the initial pixel depth feature; The performing depth feature detection on the super-resolved image data at multiple scales to obtain super-resolved depth features at multiple scales includes: Performing depth detection on the super-resolution image data to obtain the super-resolution pixel depth feature; Based on the feature distribution of the super-resolved pixel depth feature, the super-resolved region depth feature is determined.

4. The method according to claim 3, characterized in that The initial pixel depth feature and the super-resolution pixel depth feature are detected by a depth estimation network; the weight of the depth estimation network is fixed, or the weight of the depth estimation network is updateable.

5. The method according to claim 3, characterized in that: The initial region depth feature includes an initial local depth feature, and the super-resolved region depth feature includes a super-resolved local depth feature; The determining the initial region depth feature based on the feature distribution of the initial pixel depth feature comprises: Dividing the depth map formed by the initial pixel depth features into blocks to obtain initial pixel depth features of each block; Performing distribution statistics on the depth values ​​of the initial pixel depth features of the blocks to obtain the initial local depth features; The determining the super-resolution region depth feature based on the feature distribution of the super-resolution pixel depth feature comprises: Dividing the depth map formed by the super-resolution pixel depth features into blocks to obtain super-resolution pixel depth features of each block; The depth values ​​of the super-resolution pixel depth features of each block are distributed and counted to obtain the super-resolution local depth features.

6. The method according to claim 3, characterized in that: The initial region depth feature includes an initial global depth feature, and the super-resolved region depth feature includes a super-resolved global depth feature; The determining the initial region depth feature based on the feature distribution of the initial pixel depth feature comprises: Counting the frequencies of the depth values ​​contained in the initial pixel depth feature to obtain the initial global depth feature; The determining the super-resolution region depth feature based on the feature distribution of the super-resolution pixel depth feature comprises: The frequencies of the depth values ​​contained in the super-resolution pixel depth feature are counted to obtain the super-resolution global depth feature.

7. The method according to claim 3, characterized in that The performing depth detection on the initial image data to obtain the initial pixel depth feature includes: Performing depth detection on the initial image data to obtain depth features to be sampled of the initial image data; According to the size of the super-resolution image data, the depth feature to be sampled is up-sampled to obtain the initial pixel depth feature.

8. The method according to claim 2, characterized in that: The initial depth features of the multiple scales include initial pixel depth features and initial local depth features, the super-resolved depth features of the multiple scales include super-resolved pixel depth features and super-resolved local depth features, and the depth loss of each scale includes pixel depth loss and local depth loss; The determining, for each of the scales, based on a difference between the initial depth feature of the first pixel and the super-resolved depth feature of the second pixel, of the depth loss of each of the scales includes: Determining the pixel depth loss based on a feature value difference between the initial pixel depth feature of the first pixel and the super-resolution pixel depth feature of the second pixel; The local depth loss is determined based on a feature value difference between the initial local depth feature of the first pixel and the super-resolved local depth feature of the second pixel.

9. The method according to claim 2, characterized in that: The initial depth features of the multiple scales include initial global depth features, the super-resolved depth features of the multiple scales include super-resolved global depth features, and the depth loss of each scale includes a global depth loss; The determining, for each of the scales, based on a difference between the initial depth feature of the first pixel and the super-resolved depth feature of the second pixel, of the depth loss of each of the scales includes: Determining a depth offset of the super-resolved global depth feature of the second pixel relative to the initial global depth feature of the first pixel; Converting the depth offset and the super-resolved global depth feature of the second pixel to obtain a global difference; Based on the global disparity, the global depth loss is determined.

10. The method according to claim 1, characterized in that The method further comprises: determining image data loss based on a pixel difference between the super-resolved image data and a reference image data, the resolution of the reference image data being greater than the resolution of the initial image data; The step of adjusting the generator model to be trained according to the depth loss includes: The generator model to be trained is adjusted according to the depth loss and the image data loss.

11. The method according to claim 10, characterized in that The method further comprises: Performing feature extraction on the super-resolution image data to obtain super-resolution image features; Performing feature extraction on the reference image data to obtain reference image features; Determining image feature loss according to a feature difference between the super-resolved image feature and the reference image feature; The step of adjusting the generator model to be trained according to the depth loss and the image data loss includes: The generator model to be trained is adjusted according to the depth loss, image data loss and image feature loss.

12. The method according to claim 1, characterized in that The step of adjusting the generator model to be trained according to the depth loss to obtain a target generator model includes: According to the depth loss, adjusting the generator model to be trained to obtain an adjusted generator model; The adjusted generator model is used to perform super-resolution processing on the image data to be discriminated, so as to obtain super-resolution image data to be discriminated; When the training stop condition is not met and the discriminator model to be trained detects that the super-resolved image data to be discriminated belongs to the reference image data, adjusting the discriminator model to be trained to obtain an adjusted discriminator model; When the training stop condition is met and the adjusted discriminator model detects that the super-resolved image data to be discriminated belongs to the reference image data, a target generator model is obtained according to the adjusted generator model.

13. The method according to claim 12, characterized in that Before the training stop condition is met, the method further includes: When the discriminator model to be trained or the adjusted discriminator model detects that the image data to be detected does not belong to the reference image data, the adjusted generator model is used as the generator model to be trained, and the step of performing super-resolution processing on the initial image data based on the generator model to be trained is returned to execute to obtain super-resolution image data.

14. The method according to claim 12, characterized in that The training stop condition includes that the number of alternating rounds reaches the number of training stop rounds; the number of alternating rounds is the number of rounds of alternating training of the generator model to be trained and the discriminator model; The number of training stop rounds is greater than the number of rounds adjusted for the adjusted discriminator model, and the number of training stop rounds is less than the number of rounds adjusted for the adjusted generator model.

15. An image processing method, characterized in that: The method comprises: Obtaining image data to be adjusted; Performing super-resolution processing on the image data to be adjusted through a target generator model to obtain target image data; Wherein, the target generator model is a generator model trained according to any one of claims 1-14.

16. A training device for an image processing model, characterized in that: The device comprises: A super-resolution module, used for performing super-resolution processing on the initial image data based on the generator model to be trained to obtain super-resolution image data; the resolution of the super-resolution image data is greater than the resolution of the initial image data; A loss module, configured to determine a depth loss of the generator model to be trained according to a depth difference between the super-resolved image data and the initial image data; A training module is used to adjust the generator model to be trained according to the depth loss to obtain a target generator model.

17. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring image data to be adjusted; A super-resolution module, used for performing super-resolution processing on the image data to be adjusted through a target generator model to obtain target image data; Wherein, the target generator model is a generator model trained according to any one of claims 1-14.

18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images