Super-resolution model training methods, super-resolution methods, and devices

Abstract

This disclosure relates to the field of computer technology, specifically to a super-resolution model training method, a super-resolution method, a super-resolution model training device, a super-resolution apparatus, a storage medium, and a device. The method includes: acquiring a low-resolution image and inputting the low-resolution image into a model to be trained; extracting features from the low-resolution image to obtain pixel features; acquiring multiple second pixel features within a preset range of the first pixel features; determining local features of the first pixel features; determining a predicted high-resolution image based on the local features of the first pixel features; and updating the neural network parameters of the model to be trained using the predicted high-resolution image and its label to obtain a super-resolution model. The technical solution of this disclosure can solve the problem of poor quality of high-resolution images obtained after super-resolution reconstruction in related technologies.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a super-resolution model training method, a super-resolution method, a super-resolution model training device, a super-resolution device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] With the rapid development of software and hardware, the application scenarios of virtual reality are becoming increasingly widespread. For example, the metaverse refers to a virtual three-dimensional space composed of multiple virtual reality worlds. Each virtual reality world can be used through computer technology to present an environment that people can see, perceive, and interact with. The metaverse integrates the real and virtual worlds and may become an important venue for digital assets and related experiences. The metaverse can be constructed using collected video image data. However, images captured by cameras typically have poor resolution, making it difficult to meet the precision requirements of the metaverse.

[0003] In related technologies, interpolation-based methods are used for super-resolution reconstruction. Specifically, interpolation algorithms are used to enlarge low-resolution images to the same size as high-resolution images. However, these methods rely solely on the simple mean of surrounding pixels to achieve super-resolution reconstruction, resulting in severe fidelity defects and poor quality of the high-resolution images obtained after super-resolution reconstruction.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a super-resolution model training method, super-resolution method, super-resolution model training device, super-resolution device, computer-readable storage medium, and electronic device, which can solve the problem of poor quality of high-resolution images obtained after super-resolution reconstruction in related technologies.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to a first aspect of this disclosure, a super-resolution model training method is provided, comprising: acquiring a low-resolution image and inputting the low-resolution image into a model to be trained; wherein the low-resolution image corresponds to a high-resolution image label; performing feature extraction on the low-resolution image to obtain low-resolution image pixel features; wherein the low-resolution image pixel features include first pixel features corresponding to each pixel; acquiring multiple second pixel features within a preset range of the first pixel features; acquiring the similarity between the first pixel features and each second pixel feature, acquiring multiple target second pixel features with a similarity greater than a preset threshold, determining local features of the first pixel features based on the multiple target second pixel features and the first pixel features; determining a predicted high-resolution image based on the local features of the first pixel features, and updating the neural network parameters of the model to be trained by the predicted high-resolution image and the high-resolution image label to obtain a super-resolution model.

[0008] In an exemplary embodiment of this disclosure, based on the aforementioned scheme, determining a predicted high-resolution image according to the local features of the first pixel features includes: obtaining high-resolution matrix features; wherein the high-resolution matrix features are used to indicate multiple high-resolution pixel coordinates in the high-resolution image; back-projecting the high-resolution pixel coordinates onto a low-resolution image to obtain high-resolution projection coordinates, determining the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image; wherein the nearest neighbor position is the position of the pixel on the low-resolution image closest to the high-resolution projection coordinates; and determining the predicted high-resolution image based on the nearest neighbor position and the local features of the first pixel features.

[0009] In an exemplary embodiment of this disclosure, based on the aforementioned scheme, determining a predicted high-resolution image according to the local features of the nearest neighbor position and the first pixel feature includes: obtaining multiple neighbor pixel features adjacent to the nearest neighbor position, performing weighted fusion on the multiple neighbor pixel features adjacent to the nearest neighbor position to obtain weighted pixel features; and determining a predicted high-resolution image according to the local features of the weighted pixel features and the first pixel feature.

[0010] In an exemplary embodiment of this disclosure, based on the aforementioned scheme, a weighted fusion of neighboring pixel features adjacent to multiple nearest neighbor positions is performed to obtain weighted pixel features, including: obtaining the area of ​​the rectangle enclosed by each neighboring pixel feature and the nearest neighbor position, and obtaining the total area composed of multiple neighboring pixel features; determining the fusion weight of each neighboring pixel feature based on the area of ​​each rectangle and the total area; and performing a weighted fusion of multiple neighboring pixel features adjacent to multiple nearest neighbor positions based on the fusion weight of each neighboring pixel feature to obtain weighted pixel features.

[0011] In an exemplary embodiment of this disclosure, based on the aforementioned scheme, determining a predicted high-resolution image according to the local features of weighted pixel features and first pixel features includes: determining the intermediate pixel value corresponding to the high-resolution pixel coordinates according to the neighboring pixel coordinates, high-resolution pixel coordinates and weighted pixel features corresponding to each neighboring pixel feature; determining the pixel value corresponding to the high-resolution pixel coordinates according to the intermediate pixel value and the local features of the first pixel feature; and determining the predicted high-resolution image according to the pixel values ​​corresponding to multiple high-resolution pixel coordinates.

[0012] According to a second aspect of this disclosure, a super-resolution method is provided, comprising: inputting a low-resolution image into a super-resolution model; wherein the super-resolution model is trained by a super-resolution model training method as described in any of the preceding claims; and obtaining a high-resolution image corresponding to the low-resolution image based on the super-resolution model.

[0013] According to a third aspect of this disclosure, a super-resolution model training apparatus is provided, comprising: a low-resolution image acquisition module for acquiring a low-resolution image and inputting the low-resolution image into a model to be trained; wherein the low-resolution image corresponds to a high-resolution image label; an image pixel feature acquisition module for extracting features from the low-resolution image to obtain low-resolution image pixel features; wherein the low-resolution image pixel features include first pixel features corresponding to each pixel; a second pixel feature acquisition module for acquiring multiple second pixel features within a preset range of the first pixel features; a local feature determination module for acquiring the similarity between the first pixel features and each second pixel feature, acquiring multiple target second pixel features with similarity greater than a preset threshold, and determining local features of the first pixel features based on the multiple target second pixel features and the first pixel features; and a model training module for determining a predicted high-resolution image based on the local features of the first pixel features, and updating the neural network parameters of the model to be trained by the predicted high-resolution image and the high-resolution image label to obtain a super-resolution model.

[0014] According to a fourth aspect of this disclosure, a super-resolution apparatus is provided, comprising: a low-resolution image input module for inputting a low-resolution image into a super-resolution model; wherein the super-resolution model is trained using a super-resolution model training method as described above; and a super-resolution reconstruction module for obtaining a high-resolution image corresponding to the low-resolution image based on the super-resolution model.

[0015] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the super-resolution model training method of the first aspect or the super-resolution method of the second aspect as described in the above embodiments.

[0016] According to a sixth aspect of this disclosure, an electronic device is provided, comprising:

[0017] One or more processors; and

[0018] A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the super-resolution model training method of the first aspect or the super-resolution method of the second aspect as described in the above embodiments.

[0019] The technical solutions provided in this disclosure may have the following beneficial effects:

[0020] In one embodiment of this disclosure, a super-resolution model training method is provided, in which a low-resolution image is acquired, input into the model to be trained, feature extraction is performed on the low-resolution image to obtain low-resolution image pixel features, multiple second pixel features within a preset range of the first pixel features are acquired, the similarity between the first pixel features and each second pixel feature is acquired, multiple target second pixel features with similarity greater than a preset threshold are acquired, local features of the first pixel feature are determined based on the multiple target second pixel features and the first pixel feature, a predicted high-resolution image is determined based on the local features of the first pixel feature, and the neural network parameters of the model to be trained are updated by the predicted high-resolution image and the high-resolution image label to obtain a super-resolution model. Through the scheme of this disclosure, the local features of a pixel can be determined based on pixels with high similarity surrounding the pixel in the low-resolution image. This expands the preset range used to determine local features with less computational resources, thereby capturing more similar target second pixel features, improving the accuracy of the local features of the first pixel feature, avoiding noisy local features, highlighting the important local features of the pixel, and performing super-resolution reconstruction based on the local features of the pixel, avoiding fidelity defects, resulting in a high-resolution image with more accurate details and higher quality after super-resolution reconstruction.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0023] Figure 1The illustration shows a schematic diagram of an exemplary system architecture for a super-resolution model training method in an exemplary embodiment of the present disclosure;

[0024] Figure 2 A flowchart illustrating a super-resolution model training method in an exemplary embodiment of this disclosure is shown schematically.

[0025] Figure 3 This schematically illustrates a flowchart of determining a predicted high-resolution image based on local features of the nearest neighbor location and a first pixel feature in an exemplary embodiment of this disclosure;

[0026] Figure 4 This schematically illustrates a flowchart of determining a predicted high-resolution image based on local features of weighted pixel features and first pixel features in an exemplary embodiment of this disclosure;

[0027] Figure 5 This schematically illustrates a flowchart of a process in an exemplary embodiment of the present disclosure, in which weighted pixel features are obtained by weighted fusion of features of multiple nearest neighboring pixels based on the fusion weights of each neighboring pixel feature;

[0028] Figure 6 This schematically illustrates a flowchart of determining a predicted high-resolution image based on pixel values ​​corresponding to multiple high-resolution pixel coordinates in an exemplary embodiment of this disclosure.

[0029] Figure 7 This schematically illustrates a flowchart of obtaining a high-resolution image corresponding to a low-resolution image based on a super-resolution model in an exemplary embodiment of this disclosure.

[0030] Figure 8 A flowchart illustrating a super-resolution method in an exemplary embodiment of this disclosure is shown schematically.

[0031] Figure 9 This schematic diagram illustrates the acquisition of weighted pixel features in a model to be trained in an exemplary embodiment of the present disclosure.

[0032] Figure 10 This schematic diagram illustrates the composition of a super-resolution model training apparatus according to an exemplary embodiment of the present disclosure.

[0033] Figure 11 This schematic diagram illustrates the composition of a super-resolution device according to an exemplary embodiment of the present disclosure;

[0034] Figure 12 The schematic diagram illustrates a structural schematic of a computer system suitable for implementing an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., may be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0037] Figure 1 A schematic diagram of an exemplary system architecture for which the super-resolution model training method of embodiments of the present disclosure can be applied is shown.

[0038] like Figure 1 As shown, system architecture 1000 may include one or more of terminal devices 1001, 1002, and 1003, network 1004, and server 1005. Network 1004 is used as a medium to provide a communication link between terminal devices 1001, 1002, and 1003 and server 1005. Network 1004 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. For example, server 1005 could be a server cluster composed of multiple servers.

[0040] Users can use terminal devices 1001, 1002, and 1003 to interact with server 1005 via network 1004 to receive or send messages, etc. Terminal devices 1001, 1002, and 1003 can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. Additionally, server 1005 can be a server providing various services.

[0041] In one embodiment, the super-resolution model training method or the execution entity of the super-resolution method disclosed herein may be a server 1005. For example, the server 1005 may acquire low-resolution images sent by terminal devices 1001, 1002, and 1003, input the low-resolution images into the model to be trained, extract features from the low-resolution images to obtain low-resolution image pixel features, acquire multiple second pixel features within a preset range of the first pixel features, acquire the similarity between the first pixel features and each second pixel feature, acquire multiple target second pixel features with similarity greater than a preset threshold, determine the local features of the first pixel features based on the multiple target second pixel features and the first pixel features, determine the predicted high-resolution image based on the local features of the first pixel features, and update the neural network parameters of the model to be trained by the predicted high-resolution image and the high-resolution image label to obtain the super-resolution model.

[0042] Furthermore, the super-resolution model training method disclosed herein can also be executed through terminal devices 1001, 1002, 1003, etc., to achieve the following: acquiring a low-resolution image, inputting the low-resolution image into the model to be trained, extracting features from the low-resolution image to obtain low-resolution image pixel features, acquiring multiple second pixel features within a preset range of the first pixel features, acquiring the similarity between the first pixel features and each second pixel feature, acquiring multiple target second pixel features with similarity greater than a preset threshold, determining the local features of the first pixel features based on the multiple target second pixel features and the first pixel features, determining the predicted high-resolution image based on the local features of the first pixel features, and updating the neural network parameters of the model to be trained by the predicted high-resolution image and the high-resolution image label to obtain the super-resolution model.

[0043] The super-resolution method disclosed herein can also be executed through terminal devices 1001, 1002, 1003, etc., to input a low-resolution image into a super-resolution model and obtain a high-resolution image corresponding to the low-resolution image based on the super-resolution model.

[0044] Furthermore, the super-resolution model training method disclosed herein can also be implemented jointly by terminal devices 1001, 1002, and 1003 and server 1005. For example, terminal devices 1001, 1002, and 1003 can acquire low-resolution images and then send the acquired high-resolution images to server 1005, so that server 1005 can input the low-resolution images into the model to be trained, perform feature extraction on the low-resolution images to obtain low-resolution image pixel features, acquire multiple second pixel features within a preset range of the first pixel features, acquire the similarity between the first pixel features and each second pixel feature, acquire multiple target second pixel features with similarity greater than a preset threshold, determine the local features of the first pixel features based on the multiple target second pixel features and the first pixel features, determine the predicted high-resolution image based on the local features of the first pixel features, and update the neural network parameters of the model to be trained by the predicted high-resolution image and the high-resolution image label to obtain the super-resolution model.

[0045] The super-resolution method disclosed herein can also be implemented jointly by terminal devices 1001, 1002, and 1003 and server 1005. For example, terminal devices 1001, 1002, and 1003 can acquire low-resolution images and then send the acquired low-resolution images to server 1005, so that server 1005 can input the low-resolution images into the super-resolution model and obtain the high-resolution images corresponding to the low-resolution images based on the super-resolution model.

[0046] With the rapid development of software and hardware, the application scenarios of virtual reality are becoming increasingly widespread. For example, the metaverse refers to a virtual three-dimensional space composed of multiple virtual reality worlds. Each virtual reality world can be used through computer technology to present an environment that people can see, perceive, and interact with. The metaverse integrates the real and virtual worlds and may become an important venue for digital assets and related experiences. The metaverse can be constructed using collected video image data. However, images captured by cameras typically have poor resolution, making it difficult to meet the precision requirements of the metaverse.

[0047] In related technologies, interpolation-based methods are used for super-resolution reconstruction. Specifically, interpolation algorithms are used to enlarge low-resolution images to the same size as high-resolution images. However, these methods rely solely on the simple mean of surrounding pixels to achieve super-resolution reconstruction, resulting in severe fidelity defects and poor quality of the high-resolution images obtained after super-resolution reconstruction.

[0048] According to the super-resolution model training method provided in this exemplary embodiment, a low-resolution image can be acquired, input into the model to be trained, feature extraction can be performed on the low-resolution image to obtain low-resolution image pixel features, multiple second pixel features within a preset range of the first pixel features can be acquired, the similarity between the first pixel features and each second pixel feature can be acquired, multiple target second pixel features with similarity greater than a preset threshold can be acquired, local features of the first pixel feature can be determined based on the multiple target second pixel features and the first pixel feature, a predicted high-resolution image can be determined based on the local features of the first pixel feature, and the neural network parameters of the model to be trained can be updated by the predicted high-resolution image and the high-resolution image label to obtain a super-resolution model. Figure 2 As shown, the super-resolution model training method may include the following steps:

[0049] Step S210: Obtain a low-resolution image and input the low-resolution image into the model to be trained; wherein, the low-resolution image corresponds to a high-resolution image label.

[0050] Step S220: Extract features from the low-resolution image to obtain low-resolution image pixel features; wherein, the low-resolution image pixel features include the first pixel features corresponding to each pixel;

[0051] Step S230: Obtain multiple second pixel features within a preset range of the first pixel features;

[0052] Step S240: Obtain the similarity between the first pixel feature and each second pixel feature, obtain multiple target second pixel features with similarity greater than a preset threshold, and determine the local features of the first pixel feature based on the multiple target second pixel features and the first pixel feature.

[0053] Step S250: Determine the predicted high-resolution image based on the local features of the first pixel feature, and update the neural network parameters of the model to be trained by using the predicted high-resolution image and the high-resolution image label to obtain the super-resolution model.

[0054] In one embodiment of this disclosure, a super-resolution model training method is provided, in which a low-resolution image is acquired, input into the model to be trained, feature extraction is performed on the low-resolution image to obtain low-resolution image pixel features, multiple second pixel features within a preset range of the first pixel features are acquired, the similarity between the first pixel features and each second pixel feature is acquired, multiple target second pixel features with similarity greater than a preset threshold are acquired, local features of the first pixel feature are determined based on the multiple target second pixel features and the first pixel feature, a predicted high-resolution image is determined based on the local features of the first pixel feature, and the neural network parameters of the model to be trained are updated by the predicted high-resolution image and the high-resolution image label to obtain a super-resolution model. Through the scheme of this disclosure, the local features of a pixel can be determined based on pixels with high similarity surrounding the pixel in the low-resolution image. This expands the preset range used to determine local features with less computational resources, thereby capturing more similar target second pixel features, improving the accuracy of the local features of the first pixel feature, avoiding noisy local features, highlighting the important local features of the pixel, and performing super-resolution reconstruction based on the local features of the pixel, avoiding fidelity defects, resulting in a high-resolution image with more accurate details and higher quality after super-resolution reconstruction.

[0055] The steps S210 to S250 of the super-resolution model training method in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.

[0056] Step S210: Obtain a low-resolution image and input the low-resolution image into the model to be trained; wherein, the low-resolution image corresponds to a high-resolution image label.

[0057] In one example embodiment of this disclosure, a low-resolution image can be acquired. Specifically, a low-resolution image refers to an image with a relatively low resolution, while a high-resolution image refers to an image with a relatively high resolution. The low-resolution image includes multiple pixels.

[0058] For example, a low-resolution image has a resolution of 128*128, while a high-resolution image has a resolution of 256*256. The low-resolution image contains 128*128 pixels.

[0059] In one example embodiment of this disclosure, a low-resolution image corresponds to a high-resolution image label, where the high-resolution image label refers to the high-resolution image corresponding to the low-resolution image.

[0060] It should be noted that this disclosure does not impose any special limitations on the resolution of low-resolution images and high-resolution images corresponding to high-resolution image labels, but the resolution of high-resolution images corresponding to high-resolution image labels is greater than that of low-resolution images.

[0061] In one example embodiment of this disclosure, when training a model to be trained, the resolution of a high-resolution image (high-resolution image label) can be reduced to obtain a low-resolution image.

[0062] For example, high-resolution images (high-resolution image labels) can be reduced to low-resolution images through image blurring or compression. Image blurring can include mean blurring, median blurring, Gaussian blurring, etc. Specifically, taking mean blurring as an example, after inputting a high-resolution image, the image can be blurred by applying an arithmetic mean to obtain a low-resolution image. Image compression can encode and compress high-resolution images to reduce the encoding rate. Image compression can be evaluated by the compression ratio, which represents the ratio of the amount of uncompressed data in the original image after conversion to the amount of data generated after compression. When performing image compression, methods such as Huffman coding, arithmetic coding, formative coding, and frequency domain coding can be used.

[0063] It should be noted that this disclosure does not impose any special restrictions on the method of acquiring low-resolution images.

[0064] In one example embodiment of this disclosure, the image may include PNG, CDR, AI, DXF, EPS, BMP, TIFF, JPEG, GIF, PSD, and other formats. It should be noted that this disclosure does not impose any special limitations on the form or format of the image.

[0065] In one example embodiment of this disclosure, a low-resolution image can be used as a model to be trained. Specifically, the model to be trained refers to a model established to complete the super-resolution reconstruction task. A super-resolution model can be obtained by training the model to be trained, thereby completing the super-resolution reconstruction task. The super-resolution reconstruction task refers to the task of reconstructing a low-resolution image into a high-resolution image.

[0066] It should be noted that this disclosure does not impose any special restrictions on the specific structure of the super-resolution model.

[0067] Step S220: Extract features from the low-resolution image to obtain the pixel features of the low-resolution image;

[0068] In one exemplary embodiment of this disclosure, after inputting a low-resolution image into the model to be trained, feature extraction can be performed on the low-resolution image to obtain low-resolution image pixel features. Specifically, feature extraction of a low-resolution image refers to extracting some key information, i.e., low-resolution image pixel features, from the low-resolution image. More specifically, feature extraction refers to converting the low-resolution image into a vector form that can be used in algorithms such as machine learning and deep learning; these vectors contain meaningful numerical descriptions related to the original data.

[0069] In one example embodiment of this disclosure, feature extraction in a super-resolution reconstruction task can be achieved by using convolutional neural networks and other deep learning models.

[0070] For example, features can be extracted from low-resolution images using convolutional neural networks. This process involves feeding the low-resolution image into the convolutional neural network and extracting feature maps from the convolutional and pooling layers.

[0071] Specifically, the low-resolution image pixel features obtained by feature extraction from the low-resolution image may include the color features of each pixel in the low-resolution image. For example, the low-resolution image pixel features can be used to indicate the values ​​of the RGB (Red Green Blue) color channels of each pixel.

[0072] The low-resolution image pixel features include the first pixel features corresponding to each pixel. Specifically, a low-resolution image contains multiple pixels. After feature extraction from the low-resolution image, the low-resolution image pixel features can be obtained, which include the first pixel features corresponding to each pixel.

[0073] Specifically, the first pixel feature can be a two-dimensional feature, a three-dimensional feature, or a multi-dimensional feature. For example, each pixel in a color image can be composed of three components: red, green, and blue, with each component ranging from 0 to 255. Therefore, for a color image, the three components of each pixel can be used as features to obtain a three-dimensional feature.

[0074] It should be noted that this disclosure does not impose any special limitations on the specific method of extracting features from low-resolution images to obtain pixel features of low-resolution images.

[0075] Step S230: Obtain multiple second pixel features within a preset range of the first pixel features;

[0076] In one exemplary embodiment of this disclosure, after obtaining the low-resolution image pixel features through the above steps, multiple second pixel features within a preset range of the first pixel features can be obtained. Specifically, the preset range of the first pixel features refers to a preset range centered on the pixel corresponding to the first pixel feature. The low-resolution image pixel features include the first pixel features corresponding to each pixel. Within the preset range of the first pixel features (the pixel features corresponding to each pixel include the first pixel features), multiple other pixel features (second pixel features, the pixel features corresponding to each pixel include multiple first pixel features) can be included, and these pixel features can be obtained.

[0077] For example, a preset area of ​​n*n can be set with the first pixel feature as the center point, and the other pixel features in the preset area other than the first pixel feature can be determined as the second pixel feature.

[0078] It should be noted that this disclosure does not impose any special limitations on the specific method of obtaining multiple second pixel features within a preset range of the first pixel features.

[0079] Step S240: Obtain the similarity between the first pixel feature and each second pixel feature, obtain multiple target second pixel features with similarity greater than a preset threshold, and determine the local features of the first pixel feature based on the multiple target second pixel features and the first pixel feature.

[0080] In one exemplary embodiment of this disclosure, after determining multiple second pixel features through the above steps, the similarity between the first pixel feature and each second pixel feature can be obtained. Specifically, the similarity between the first pixel feature and each second pixel feature can be determined using a similarity measurement method. The similarity between the first pixel feature and each second pixel feature can be used to indicate whether the first pixel feature and each second pixel feature have similar attributes or features.

[0081] For example, the similarity between the first pixel feature and each second pixel feature can be determined by a distance metric.

[0082] It should be noted that this disclosure does not impose any special limitations on the specific method for obtaining the similarity between the first pixel feature and each second pixel feature.

[0083] In one exemplary embodiment of this disclosure, after determining the similarity between a first pixel feature and each second pixel feature, the relationship between each similarity and a preset threshold can be determined. When the similarity between a second pixel feature and a first pixel feature is greater than the preset threshold, the second pixel feature can be identified as a target second pixel feature, and these target second pixel features can be fused when determining the local features of the first pixel feature. Conversely, when the similarity between a second pixel feature and a first pixel feature is less than or equal to the preset threshold, these second pixel features are not fused when determining the local features of the first pixel feature. Through the solution of this disclosure, noisy features can be reduced when determining local features, highlighting the salient features of the first pixel feature, thereby improving the super-resolution effect.

[0084] In one exemplary embodiment of this disclosure, the similarity between a first pixel feature and each of the second pixel features can be determined by the following expression, where X i X is the feature of the first pixel. j For the second pixel feature, Wθ and Wψ are convolution kernels of size 1*1, φ(X) i X j ) is X i With X j Similarity:

[0085] φ(X i X j )=exp{X i WθWψ T X j T}

[0086] After obtaining the similarity between the first pixel feature and each second pixel feature, the similarity between the first pixel feature and each second pixel feature can be compared with a preset threshold k. If the similarity between the first pixel feature and the second pixel feature is greater than k, then the second pixel feature is determined as the target second pixel feature. If the similarity between the first pixel feature and the second pixel feature is less than or equal to k, then the similarity between the second pixel feature and the first pixel feature is marked as 0. When determining the local features of the first pixel feature, these second pixel features with similarity marked as 0 are not fused. Through the scheme disclosed herein, the preset range used to determine local features can be expanded with less computing resources, thereby capturing more target second pixel features with higher similarity, thus improving the accuracy of the local features of the first pixel feature, making the details of the high-resolution image obtained after super-resolution reconstruction more accurate and the quality of the high-resolution image higher.

[0087] In one exemplary embodiment of this disclosure, after obtaining the target second pixel feature through the above steps, the local features of the first pixel feature can be determined based on multiple target second pixel features and the first pixel feature. Specifically, multiple target second pixel features and the first pixel feature can be fused to determine the local features of the first pixel feature.

[0088] In one exemplary embodiment of this disclosure, the expression for determining the local features of the first pixel feature based on multiple target second pixel features and first pixel features is as follows:

[0089]

[0090] Among them, y i X is a local feature of the first pixel. i X is the feature of the first pixel. j For the j-th second pixel feature, φ(X) i X j ) is X i With X j Similarity between Wθ, Wψ and W g These are convolution kernels of size 1*1, S i It is a set of second pixel features of multiple targets.

[0091] It should be noted that this disclosure does not impose any special limitations on the specific method of determining the local features of the first pixel feature based on the second pixel features and the first pixel features of multiple targets.

[0092] Step S250: Determine the predicted high-resolution image based on the local features of the first pixel feature, and update the neural network parameters of the model to be trained by using the predicted high-resolution image and the high-resolution image label to obtain the super-resolution model.

[0093] In one exemplary embodiment of this disclosure, after obtaining the local features of the first pixel feature through the above steps, a predicted high-resolution image can be determined based on the local features of the first pixel feature. Specifically, the local features of the pixel features corresponding to each pixel in the low-resolution image pixel features can be determined according to the above steps, and the predicted high-resolution image can be determined based on the local features of the pixel features corresponding to each pixel.

[0094] In one example embodiment of this disclosure, the model to be trained may include multiple hidden layers, which may include convolutional layers, normalization layers, activation layers, etc. A low-resolution image can be input into the multiple hidden layers of the model to be trained to obtain the hidden layer calculation results (including local features of each pixel). The predicted high-resolution image can then be obtained through the hidden layer calculation results.

[0095] It should be noted that this disclosure does not impose any special limitations on the specific method of determining the predicted high-resolution image based on the local features of the first pixel features.

[0096] In one example embodiment of this disclosure, after obtaining the predicted high-resolution image through the above steps, the neural network parameters of the model to be trained can be updated based on the high-resolution image label and the predicted high-resolution image to obtain a super-resolution model. Specifically, the predicted high-resolution image can be used to indicate the predicted high-resolution image obtained after super-resolution reconstruction of the low-resolution image. The predicted high-resolution image is a predicted value, and the corresponding true value, i.e., the high-resolution image label, can be obtained. This high-resolution image label can be used to indicate the true high-resolution image obtained after super-resolution reconstruction of the low-resolution image. At this time, the predicted high-resolution image (predicted value) can be compared with the high-resolution image label (true value) to obtain the prediction difference between the predicted high-resolution image (predicted value) and the high-resolution image label (true value), and the neural network parameters of the model to be trained can be updated based on this prediction difference to obtain a super-resolution model.

[0097] Specifically, the neural network parameters of the model to be trained may include the number of model layers, the number of feature vector channels, and the learning rate. When updating the neural network parameters of the model to be trained based on the prediction difference, the number of model layers, the number of feature vector channels, and the learning rate of the model to be trained can be updated to train the super-resolution model.

[0098] In one example embodiment of this disclosure, the neural network parameters of the model to be trained can be updated using the backpropagation algorithm, and a super-resolution model can be obtained after training.

[0099] It should be noted that this disclosure does not impose any special limitations on the specific method of updating the neural network parameters of the model to be trained based on the high-resolution image labels and the predicted high-resolution images.

[0100] In one exemplary embodiment of this disclosure, the neural network parameters of the model to be trained can be updated based on the high-resolution image labels and the predicted high-resolution image. When the model to be trained meets the convergence condition, it is determined to be a super-resolution model. Specifically, meeting the convergence condition means that the model to be trained has high prediction accuracy and can be applied. For example, the convergence condition may include the number of training iterations, such as ending training after the model to be trained N times; or the convergence condition may include the training duration, such as ending training after the model to be trained for a duration T.

[0101] It should be noted that this disclosure does not impose any special limitations on the specific content of the convergence conditions. By applying convergence conditions to the model, the training process of the model to be trained can be better controlled, the problem of overtraining of neural networks can be avoided, and the training efficiency of the model to be trained can be improved.

[0102] In one example embodiment of this disclosure, high-resolution matrix features can be obtained, high-resolution pixel coordinates can be back-projected onto a low-resolution image to obtain high-resolution projected coordinates, the nearest neighbor position of the high-resolution projected coordinates on the low-resolution image can be determined, and the predicted high-resolution image can be determined based on the nearest neighbor position and the local features of the first pixel feature. (Refer to...) Figure 3 As shown, determining the predicted high-resolution image based on the local features of the nearest neighbor position and the first pixel feature may include the following steps S310 to S330:

[0103] Step S310: Obtain high-resolution matrix features;

[0104] In one example embodiment of this disclosure, high-resolution matrix features can be obtained. These high-resolution matrix features are used to indicate the coordinates of multiple high-resolution pixels in a high-resolution image. Specifically, high-resolution matrix features refer to the positional features of the high-resolution image obtained after super-resolution reconstruction of a low-resolution image. The high-resolution matrix features can indicate the position of each pixel in the high-resolution image obtained after super-resolution reconstruction of the low-resolution image, i.e., the coordinates of multiple high-resolution pixels in the high-resolution image.

[0105] It should be noted that this disclosure does not impose any special limitations on the specific method for obtaining high-resolution matrix features.

[0106] Step S320: Back-project the high-resolution pixel coordinates onto the low-resolution image to obtain high-resolution projection coordinates, and determine the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image.

[0107] In one example embodiment of this disclosure, after obtaining the high-resolution matrix features through the above steps, the high-resolution pixel coordinates can be back-projected onto the low-resolution image to obtain high-resolution projected coordinates. Specifically, the coordinates of a pixel on the high-resolution image (high-resolution pixel coordinates) can be mapped onto the low-resolution image through back-projection, and the position of the high-resolution pixel coordinates back-projected onto the low-resolution image (high-resolution projected coordinates) can be obtained.

[0108] In one exemplary embodiment of this disclosure, after determining the high-resolution projection coordinates, the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image can be determined. The nearest neighbor position is the position of the pixel on the low-resolution image closest to the high-resolution projection coordinates. Specifically, when the high-resolution projection coordinates are located at the position of a pixel on a low-resolution image, the position of that pixel on the low-resolution image can be determined as the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image; when the high-resolution projection coordinates are not located at the position of a pixel on a low-resolution image, the position of the pixel closest to the high-resolution projection coordinates can be found on the low-resolution image, and that pixel's position can be determined as the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image.

[0109] It should be noted that this disclosure does not impose any special limitations on the specific method for determining the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image.

[0110] Step S330: Determine the predicted high-resolution image based on the local features of the nearest neighbor position and the first pixel feature.

[0111] In one exemplary embodiment of this disclosure, after obtaining the nearest neighbor positions of the high-resolution projected coordinates on the low-resolution image through the above steps, the predicted high-resolution image can be determined based on the nearest neighbor positions and the local features of the first pixel feature. Specifically, the nearest neighbor positions of each high-resolution pixel coordinate in the high-resolution image on the low-resolution image can be obtained, the local features of each pixel feature can be determined according to the above steps, and the predicted high-resolution image can be determined based on the nearest neighbor positions of multiple high-resolution pixel coordinates on the low-resolution image and the local features of each pixel feature.

[0112] It should be noted that this disclosure does not impose any special limitations on the specific method of determining the predicted high-resolution image based on the local features of the nearest neighbor position and the features of the first pixel.

[0113] Through the above steps S310 to S330, high-resolution matrix features can be obtained, high-resolution pixel coordinates can be back-projected onto the low-resolution image to obtain high-resolution projection coordinates, the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image can be determined, and the predicted high-resolution image can be determined based on the nearest neighbor position and the local features of the first pixel feature.

[0114] In one example embodiment of this disclosure, multiple nearest neighbor pixel features can be obtained, and weighted fusion of these features is performed to obtain weighted pixel features. A predicted high-resolution image is then determined based on the local features of the weighted pixel features and the first pixel features. (Refer to...) Figure 4As shown, determining the predicted high-resolution image based on the local features of the weighted pixel features and the first pixel features may include the following steps S410 to S420:

[0115] Step S410: Obtain the neighboring pixel features of multiple nearest neighbor positions, and perform weighted fusion on the neighboring pixel features of multiple nearest neighbor positions to obtain weighted pixel features.

[0116] In one example embodiment of this disclosure, after obtaining the nearest neighbor position of the high-resolution pixel coordinates on the low-resolution image through the above steps, the neighboring pixel features adjacent to the nearest neighbor position can be obtained. Specifically, the neighboring pixel features adjacent to the nearest neighbor position refer to the pixel features in the low-resolution image that are adjacent to the nearest neighbor position.

[0117] It should be noted that this disclosure does not impose any special limitation on the number of neighboring pixel features.

[0118] In one exemplary embodiment of this disclosure, after obtaining the features of multiple neighboring pixels at their nearest neighbor positions, a weighted fusion of these features can be performed to obtain a weighted pixel feature. Specifically, performing a weighted fusion of the features of multiple neighboring pixels at their nearest neighbor positions to obtain a weighted pixel feature may include, for the nearest position, performing a weighted fusion of the features of multiple neighboring pixels surrounding it, thereby obtaining a feature representation of the fused feature. The weight of each neighboring pixel feature can be determined, and the weighted fusion of the features of multiple neighboring pixels at their nearest neighbor positions can be performed according to the weight of each neighboring pixel feature to obtain the weighted pixel feature.

[0119] For example, the weights of each neighboring pixel feature can be determined based on factors such as distance or similarity between different pixels, and the weights of each neighboring pixel feature can be used to weight and fuse the features of multiple nearest neighboring pixels to obtain weighted pixel features.

[0120] It should be noted that this disclosure does not impose any special limitations on the method of weighted fusion of the features of multiple nearest neighbor pixels to obtain weighted pixel features.

[0121] Step S420: Determine the predicted high-resolution image based on the local features of the weighted pixel features and the first pixel features.

[0122] In one exemplary embodiment of this disclosure, a predicted high-resolution image can be determined based on the local features of weighted pixel features and first pixel features. Specifically, the nearest neighbor positions of each high-resolution pixel coordinate in the high-resolution image on the low-resolution image can be obtained through the above steps, and multiple neighboring pixel features adjacent to each nearest neighbor position can be obtained to determine the weighted pixel features corresponding to each nearest neighbor position. The local features of each pixel feature can be determined according to the above steps, so as to determine the predicted high-resolution image based on the local features of each pixel feature (first pixel feature) and the weighted pixel features corresponding to each nearest neighbor position.

[0123] It should be noted that this disclosure does not impose any special limitations on the specific method of determining the predicted high-resolution image based on the local features of the weighted pixel features and the first pixel features.

[0124] Through the above steps S410 to S420, the features of neighboring pixels at multiple nearest neighbor positions can be obtained. The features of neighboring pixels at multiple nearest neighbor positions are weighted and fused to obtain weighted pixel features. The predicted high-resolution image is determined based on the local features of the weighted pixel features and the first pixel features.

[0125] In one example embodiment of this disclosure, the area of ​​the rectangles formed by each neighboring pixel feature and the nearest neighbor position can be obtained, the total area formed by multiple neighboring pixel features can be obtained, the fusion weight of each neighboring pixel feature can be determined based on the area of ​​each rectangle and the total area, and the neighboring pixel features adjacent to the nearest neighbor position can be weighted and fused according to the fusion weight of each neighboring pixel feature to obtain a weighted pixel feature. (Refer to...) Figure 5 As shown, the weighted pixel features are obtained by weighting and fusing the features of multiple nearest neighbor pixels according to the fusion weights of each neighbor pixel feature, which may include the following steps S510 to S520:

[0126] Step S510: Obtain the area of ​​the rectangle formed by each neighboring pixel feature and the nearest position, and obtain the total area formed by multiple neighboring pixel features;

[0127] In one example embodiment of this disclosure, after obtaining the nearest position of the high-resolution projection coordinates on the low-resolution image through the above steps, and determining the neighboring pixel features adjacent to the nearest position, the area of ​​the rectangle enclosed by each neighboring pixel feature and the nearest position can be obtained, and the total area composed of multiple neighboring pixel features can be obtained. Specifically, the nearest position corresponds to a coordinate, and each neighboring pixel feature corresponds to a coordinate. The coordinates corresponding to the nearest position and the coordinates corresponding to each neighboring pixel feature can be used as the two endpoints of the diagonal of the rectangle, and the area of ​​the rectangle enclosed by the neighboring pixel features and each nearest position can be determined accordingly. The coordinates corresponding to multiple neighboring pixel features are then connected to obtain a shape, and the total area of ​​the shape is obtained. The total area of ​​the shape is the total area composed of multiple neighboring pixel features.

[0128] Step S520: Determine the fusion weight of each neighboring pixel feature based on the area of ​​each rectangle and the total area, and perform weighted fusion on the neighboring pixel features of multiple nearest neighbor positions based on the fusion weight of each neighboring pixel feature to obtain weighted pixel features.

[0129] In one exemplary embodiment of this disclosure, after obtaining the rectangular area enclosed by each neighboring pixel feature and the nearest neighbor position, and the total area formed by multiple neighboring pixel features through the above steps, the fusion weight of each neighboring pixel feature can be determined based on the rectangular area and the total area. Specifically, the ratio of the rectangular area to the total area can be used as the fusion weight of the neighboring pixel feature corresponding to the rectangular area, and the neighboring pixel features adjacent to multiple nearest neighbor positions are weighted and fused according to the fusion weight of each neighboring pixel feature to obtain a weighted pixel feature.

[0130] It should be noted that this disclosure does not impose any special limitations on the specific method for determining the fusion weights of each neighboring pixel feature based on the area of ​​each rectangle and the total area.

[0131] In one exemplary embodiment of this disclosure, a weighted pixel feature is obtained by weighted fusing the features of multiple nearest neighbor pixels based on the fusion weights of each neighbor pixel feature. The expression for this weighted pixel feature is as follows:

[0132]

[0133] Among them, I( P ) represents the weighted pixel features, f(Z) i (00, 01, 10, 11) represent the features of neighboring pixels, and (00, 01, 10, 11) are used to indicate the features of the four nearest neighboring pixels. i Let S be the area of ​​the rectangle formed by the i-th neighboring pixel feature and the nearest neighbor position, and let S be the total area formed by multiple neighboring pixel features.

[0134] Compared to existing super-resolution models that require a fixed super-resolution reconstruction factor, the solution in this embodiment does not require a fixed reconstruction factor. Instead, it constructs the pixel value of each pixel in the high-resolution image through implicit prediction, which is not limited by the sampling factor. This allows for super-resolution reconstruction tasks at any scale, including non-integer scales, thus improving the versatility of super-resolution reconstruction tasks.

[0135] Through the above steps S510 to S520, the area of ​​the rectangle formed by each neighboring pixel feature and the nearest position can be obtained, the total area formed by multiple neighboring pixel features can be obtained, the fusion weight of each neighboring pixel feature can be determined according to the area of ​​each rectangle and the total area, and the weighted fusion of multiple neighboring pixel features adjacent to the nearest position can be performed according to the fusion weight of each neighboring pixel feature to obtain the weighted pixel feature.

[0136] In one exemplary embodiment of this disclosure, the intermediate pixel value corresponding to the high-resolution pixel coordinates can be determined based on the neighboring pixel coordinates, high-resolution pixel coordinates, and weighted pixel features corresponding to each neighboring pixel feature. The pixel value corresponding to the high-resolution pixel coordinates can be determined based on the intermediate pixel value and the local features of the first pixel feature. The predicted high-resolution image can be determined based on the pixel values ​​corresponding to multiple high-resolution pixel coordinates. (Refer to...) Figure 6 As shown, determining the predicted high-resolution image based on the pixel values ​​corresponding to multiple high-resolution pixel coordinates may include the following steps S610 to S630:

[0137] Step S610: Determine the intermediate pixel value corresponding to the high-resolution pixel coordinates based on the neighboring pixel coordinates, high-resolution pixel coordinates, and weighted pixel features corresponding to each neighboring pixel feature.

[0138] In one example embodiment of this disclosure, after obtaining the neighboring pixel coordinates corresponding to the neighboring pixel features through the above steps, the intermediate pixel value corresponding to the high-resolution pixel coordinates can be determined based on the neighboring pixel coordinates corresponding to each neighboring pixel feature, the high-resolution pixel coordinates, and the weighted pixel features. Specifically, the high-resolution pixel coordinates refer to the coordinates of pixels in the high-resolution image after super-resolution reconstruction of pixels in the low-resolution pixel image. The nearest position of the high-resolution projection coordinates on the low-resolution image can be obtained, and multiple neighboring pixel features adjacent to the nearest position can be obtained. Each neighboring pixel feature corresponds to a neighboring pixel coordinate. The intermediate pixel value corresponding to each high-resolution pixel coordinate can be determined by combining the neighboring pixel coordinates corresponding to each neighboring pixel feature, the high-resolution pixel coordinates, and the weighted pixel features. This intermediate pixel value can be used to indicate the RGB value of the high-resolution pixel.

[0139] In one example embodiment of this disclosure, the intermediate pixel value corresponding to the high-resolution pixel coordinates can be determined by an implicit function, the expression of which is as follows:

[0140]

[0141] Among them, I( Q Z represents the intermediate pixel value corresponding to the high-resolution pixel coordinates, f() is an implicit function, and Z... i X is the feature of the i-th neighboring pixel. q v is the nearest neighbor. i S represents the coordinates of the nearest neighbor pixel corresponding to the nearest pixel feature, c represents the high-resolution pixel coordinates, and (00, 01, 10, 11) are used to indicate the four nearest neighbor pixel features at the nearest position, respectively. i Let S be the area of ​​the rectangle formed by the i-th neighboring pixel feature and the nearest neighbor position, and let S be the total area formed by multiple neighboring pixel features.

[0142] Step S620: Determine the pixel value corresponding to the high-resolution pixel coordinates based on the intermediate pixel value and the local features of the first pixel feature;

[0143] In one exemplary embodiment of this disclosure, the pixel value corresponding to the high-resolution pixel coordinates can be determined based on the intermediate pixel value and the local features of the first pixel feature. Specifically, the intermediate pixel value (the intermediate pixel value corresponding to the multiple high-resolution pixel coordinates) of multiple pixels in the high-resolution image can be determined through the above steps, and the local features of each pixel feature (the first pixel feature) can be determined according to the above steps, so as to determine the pixel value corresponding to the multiple high-resolution pixel coordinates based on the intermediate pixel value corresponding to multiple pixels in the multiple high-resolution image and the local features of the multiple pixel features (the first pixel feature).

[0144] Step S630: Determine the predicted high-resolution image based on the pixel values ​​corresponding to multiple high-resolution pixel coordinates.

[0145] In one exemplary embodiment of this disclosure, after determining the pixel values ​​corresponding to multiple high-resolution pixel coordinates, a predicted high-resolution image can be determined based on the pixel values ​​corresponding to the multiple high-resolution pixel coordinates. Specifically, the pixel values ​​of each pixel in the predicted high-resolution image can be determined using the pixel values ​​corresponding to the multiple high-resolution pixel coordinates, and the predicted high-resolution image can be determined using multiple pixels with pixel values.

[0146] Through the above steps S610 to S630, the intermediate pixel value corresponding to the high-resolution pixel coordinate can be determined based on the neighboring pixel coordinates, high-resolution pixel coordinates, and weighted pixel features corresponding to each neighboring pixel feature. The pixel value corresponding to the high-resolution pixel coordinate can be determined based on the intermediate pixel value and the local features of the first pixel feature. The predicted high-resolution image can be determined based on the pixel values ​​corresponding to multiple high-resolution pixel coordinates.

[0147] In one example embodiment of this disclosure, a low-resolution image can be input into a super-resolution model, and a high-resolution image corresponding to the low-resolution image can be obtained based on the super-resolution model. (Refer to...) Figure 7 As shown, obtaining the high-resolution image corresponding to the low-resolution image based on the super-resolution model may include the following steps S710 to S720:

[0148] Step S710: Input the low-resolution image into the super-resolution model; wherein the super-resolution model is trained by any of the super-resolution model training methods described above.

[0149] Step S720: Obtain the high-resolution image corresponding to the low-resolution image based on the super-resolution model.

[0150] In one example embodiment of this disclosure, when a low-resolution image is obtained, the low-resolution image can be input into a super-resolution model, and the super-resolution model can regenerate the corresponding high-resolution image based on the low-resolution image.

[0151] Through steps S710-S720, a low-resolution image can be input into a super-resolution model, and a high-resolution image corresponding to the low-resolution image can be obtained based on the super-resolution model. The super-resolution model can make the low-resolution image clearer. Based on this, pixels with high similarity to pixels in the low-resolution image are used to determine the local features of that pixel, avoiding noise in local features and highlighting the important local features of the pixel. Super-resolution reconstruction is then performed based on these local features, avoiding fidelity defects and resulting in a high-resolution image with more accurate details and higher quality.

[0152] In one exemplary embodiment of this disclosure, such as Figure 8 The diagram illustrates a super-resolution method, which may include the following steps S801–S810:

[0153] Step S801: Acquire a low-resolution image;

[0154] Step S802, Feature extraction; Perform feature extraction on the low-resolution image to obtain the pixel features of the low-resolution image;

[0155] Step S803, Local feature fusion; Obtain multiple second pixel features within a preset range of the first pixel feature, obtain the similarity between the first pixel feature and each second pixel feature, obtain multiple target second pixel features with similarity greater than a preset threshold, and determine the local features of the first pixel feature based on the multiple target second pixel features and the first pixel feature;

[0156] Step S804: Obtain high-resolution matrix features;

[0157] Step S805: Determine the nearest neighbor position; back-project the high-resolution pixel coordinates onto the low-resolution image to obtain high-resolution projection coordinates, and determine the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image.

[0158] Step S806, Feature fusion; fusing local features of pixel features with the nearest neighbor location;

[0159] Step S807: Obtain neighboring pixel features; obtain the features of neighboring pixels adjacent to multiple nearest neighbor positions;

[0160] Step S808: Weighted pixel features are obtained by weighted fusion of neighboring pixel features; weighted pixel features are obtained by weighted fusion of neighboring pixel features at multiple nearest neighbor positions.

[0161] Step S809, Implicit prediction: Based on the neighboring pixel coordinates, high-resolution pixel coordinates and weighted pixel feature polarity of each neighboring pixel feature, implicit prediction is performed to determine the intermediate pixel value corresponding to the high-resolution pixel coordinates.

[0162] Step S810: Obtain the predicted high-resolution image; determine the pixel value corresponding to the high-resolution pixel coordinates based on the intermediate pixel value and the local features of the first pixel feature; determine the predicted high-resolution image based on the pixel values ​​corresponding to multiple high-resolution pixel coordinates.

[0163] In one exemplary embodiment of this disclosure, such as Figure 9 The diagram shown illustrates the process of obtaining weighted pixel features in the model to be trained.

[0164] The process involves several steps: inputting a low-resolution image into the model to be trained; extracting low-resolution image pixel features 901; obtaining multiple second pixel features within a preset range of the first pixel features 902; obtaining the similarity between the first pixel features and each second pixel feature; obtaining multiple target second pixel features with similarity greater than a preset threshold k; determining the local features of the pixel feature (first pixel feature) based on the multiple target second pixel features and the first pixel feature; back-projecting high-resolution pixel coordinates onto the low-resolution image to obtain high-resolution projection coordinates; determining the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image 903; fusing the local features of the pixel feature with the nearest neighbor position to obtain fused features 904; fusing the nearest neighbor position corresponding to each high-resolution pixel coordinate with the local features of the pixel feature to obtain the total fused features 905; obtaining the neighboring pixel features adjacent to multiple nearest neighbor positions 906; and obtaining each neighboring pixel feature (Z...). 00 Z 01 Z 10 Z 11 ) and nearest neighbor x qThe area of ​​the rectangle is obtained, and the total area formed by multiple neighboring pixel features is obtained. The fusion weight of each neighboring pixel feature is determined based on the area of ​​each rectangle and the total area. The weighted fusion of multiple neighboring pixel features with the nearest neighbor position is obtained by weighted fusion based on the fusion weight of each neighboring pixel feature.

[0165] In one embodiment of this disclosure, a super-resolution model training method is provided, in which a low-resolution image is acquired, input into the model to be trained, feature extraction is performed on the low-resolution image to obtain low-resolution image pixel features, multiple second pixel features within a preset range of the first pixel features are acquired, the similarity between the first pixel features and each second pixel feature is acquired, multiple target second pixel features with similarity greater than a preset threshold are acquired, local features of the first pixel feature are determined based on the multiple target second pixel features and the first pixel feature, a predicted high-resolution image is determined based on the local features of the first pixel feature, and the neural network parameters of the model to be trained are updated by the predicted high-resolution image and the high-resolution image label to obtain a super-resolution model. Through the scheme of this disclosure, the local features of a pixel can be determined based on pixels with high similarity surrounding the pixel in the low-resolution image. This expands the preset range used to determine local features with less computational resources, thereby capturing more similar target second pixel features, improving the accuracy of the local features of the first pixel feature, avoiding noisy local features, highlighting the important local features of the pixel, and performing super-resolution reconstruction based on the local features of the pixel, avoiding fidelity defects, resulting in a high-resolution image with more accurate details and higher quality after super-resolution reconstruction.

[0166] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0167] Furthermore, in an exemplary embodiment of this disclosure, a super-resolution model training apparatus is also provided. (Refer to...) Figure 10 As shown, a super-resolution model training device 10000 includes: a low-resolution image acquisition module 1010, an image pixel feature acquisition module 1020, a second pixel feature acquisition module 1030, a local feature determination module 1040, and a model training module 1050.

[0168] The system includes the following modules: a low-resolution image acquisition module for acquiring low-resolution images and inputting them into the model to be trained; each low-resolution image corresponds to a high-resolution image label; an image pixel feature acquisition module for extracting features from the low-resolution image to obtain low-resolution image pixel features; each low-resolution image pixel feature includes a first pixel feature corresponding to each pixel; a second pixel feature acquisition module for acquiring multiple second pixel features within a preset range of the first pixel features; a local feature determination module for acquiring the similarity between the first pixel features and each second pixel feature, acquiring multiple target second pixel features with a similarity greater than a preset threshold, and determining the local features of the first pixel features based on the multiple target second pixel features and the first pixel features; and a model training module for determining the predicted high-resolution image based on the local features of the first pixel features, updating the neural network parameters of the model to be trained using the predicted high-resolution image and the high-resolution image label to obtain a super-resolution model.

[0169] In an exemplary embodiment of this disclosure, based on the aforementioned scheme, the apparatus further includes: a high-resolution matrix feature acquisition unit, configured to acquire high-resolution matrix features; wherein the high-resolution matrix features are used to indicate multiple high-resolution pixel coordinates in the high-resolution image; a back-projection unit, configured to back-project the high-resolution pixel coordinates onto a low-resolution image to obtain high-resolution projection coordinates, and determine the nearest neighbor position of the high-resolution projection coordinates on the low-resolution image; wherein the nearest neighbor position is the position of the pixel closest to the high-resolution projection coordinates on the low-resolution image; and a first high-resolution image prediction unit, configured to determine the predicted high-resolution image based on the nearest neighbor position and the local features of the first pixel feature.

[0170] In an exemplary embodiment of this disclosure, based on the aforementioned scheme, the apparatus further includes: a neighboring pixel feature acquisition unit, configured to acquire neighboring pixel features adjacent to a plurality of neighboring positions, and to perform weighted fusion on the neighboring pixel features adjacent to the plurality of neighboring positions to obtain weighted pixel features; and a second high-resolution image prediction unit, configured to determine the predicted high-resolution image based on the weighted pixel features and the local features of the first pixel feature.

[0171] In an exemplary embodiment of this disclosure, based on the aforementioned scheme, a weighted pixel feature is obtained by weighted fusion of multiple neighboring pixel features adjacent to the nearest neighbor position. The apparatus further includes: a rectangular area acquisition unit, used to acquire the rectangular area enclosed by each neighboring pixel feature and the nearest neighbor position, and to acquire the total area composed of multiple neighboring pixel features; and a weight determination unit, used to determine the fusion weight of each neighboring pixel feature based on the rectangular area and the total area, and to perform weighted fusion of multiple neighboring pixel features adjacent to the nearest neighbor position based on the fusion weight of each neighboring pixel feature to obtain the weighted pixel feature.

[0172] In an exemplary embodiment of this disclosure, based on the aforementioned scheme, a predicted high-resolution image is determined according to the local features of the weighted pixel features and the first pixel features. The apparatus further includes: an intermediate pixel value determination unit, configured to determine the intermediate pixel value corresponding to the high-resolution pixel coordinates according to the neighboring pixel coordinates, the high-resolution pixel coordinates, and the weighted pixel features corresponding to each neighboring pixel feature; a pixel value determination unit, configured to determine the pixel value corresponding to the high-resolution pixel coordinates according to the intermediate pixel value and the local features of the first pixel features; and a third high-resolution image prediction unit, configured to determine the predicted high-resolution image according to the pixel values ​​corresponding to the multiple high-resolution pixel coordinates.

[0173] Since the functional modules of the super-resolution model training apparatus in the example embodiments of this disclosure correspond to the steps of the example embodiments of the super-resolution model training method described above, for details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the super-resolution model training method described above.

[0174] Furthermore, in an exemplary embodiment of this disclosure, a super-resolution device is also provided. (Refer to...) Figure 11 As shown, a super-resolution device 1100 includes: a low-resolution image input module 1110 and a super-resolution reconstruction module 1120.

[0175] The low-resolution image input module is used to input the low-resolution image into the super-resolution model; the super-resolution model is trained using any of the super-resolution model training methods mentioned above; the super-resolution reconstruction module is used to obtain the high-resolution image corresponding to the low-resolution image based on the super-resolution model.

[0176] Since the functional modules of the super-resolution apparatus in the example embodiments of this disclosure correspond to the steps of the example embodiments of the super-resolution method described above, for details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the super-resolution method described above.

[0177] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0178] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described super-resolution model training method or super-resolution method is also provided.

[0179] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be embodied in the following forms: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0180] The following reference Figure 12 To describe an electronic device 1200 according to such an embodiment of the present disclosure. Figure 12 The electronic device 1200 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0181] like Figure 12 As shown, the electronic device 1200 is manifested in the form of a general-purpose computing device. The components of the electronic device 1200 may include, but are not limited to: at least one processing unit 1210, at least one storage unit 1220, a bus 1230 connecting different system components (including storage unit 1220 and processing unit 1210), and a display unit 1240.

[0182] The storage unit stores program code, which can be executed by the processing unit 1210 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1210 can perform actions such as... Figure 2Step S210, as shown, involves acquiring a low-resolution image and inputting it into the model to be trained; wherein the low-resolution image corresponds to a high-resolution image label; Step S220 involves extracting features from the low-resolution image to obtain low-resolution image pixel features; wherein the low-resolution image pixel features include the first pixel features corresponding to each pixel; Step S230 involves acquiring multiple second pixel features within a preset range of the first pixel features; Step S240 involves acquiring the similarity between the first pixel features and each second pixel feature, acquiring multiple target second pixel features with similarity greater than a preset threshold, and determining the local features of the first pixel features based on the multiple target second pixel features and the first pixel features; Step S250 involves determining the predicted high-resolution image based on the local features of the first pixel features, and updating the neural network parameters of the model to be trained by predicting the high-resolution image and the high-resolution image label to obtain a super-resolution model. For example, processing unit 1210 can perform the following... Figure 7 In step S710, the low-resolution image is input into the super-resolution model; wherein the super-resolution model is trained by any of the super-resolution model training methods described above; in step S720, the high-resolution image corresponding to the low-resolution image is obtained based on the super-resolution model.

[0183] Storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 1221 and / or cache memory 1222, and may further include a read-only memory (ROM) 1223.

[0184] Storage unit 1220 may also include a program / utility 1224 having a set (at least one) of program modules 1225, such program modules 1225 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0185] Bus 1230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration open port, a processing unit, or a local bus using any of the multiple bus structures.

[0186] Electronic device 1200 can also communicate with one or more external devices 1270 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 1200, and / or with any device that enables electronic device 1200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1260. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0187] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0188] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0189] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0190] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0191] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0192] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0193] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A method for training a super-resolution model, characterized in that, The method comprises: acquiring a low-resolution image, inputting the low-resolution image into a to-be-trained model; wherein the low-resolution image corresponds to a high-resolution image label; the to-be-trained model is a neural network model used for performing a super-resolution reconstruction task; performing feature extraction on the low-resolution image to obtain low-resolution image pixel features; wherein the low-resolution image pixel features include first pixel features corresponding to each pixel; acquiring a plurality of second pixel features within a preset range of the first pixel feature; acquiring the similarity of the first pixel feature and each second pixel feature, acquiring a plurality of target second pixel features with a similarity greater than a preset threshold, and determining a local feature of the first pixel feature according to the plurality of target second pixel features and the first pixel feature; determining a predicted high-resolution image according to the local feature of the first pixel feature, updating the neural network parameters of the to-be-trained model through the predicted high-resolution image and the high-resolution image label, to obtain a super-resolution model.

2. The method of claim 1, wherein, The method comprises: acquiring a high-resolution matrix feature; wherein the high-resolution matrix feature is used to indicate a plurality of high-resolution pixel coordinates in the high-resolution image; back-projecting the high-resolution pixel coordinates to a low-resolution image to obtain high-resolution projection coordinates, and determining the nearest position of the high-resolution projection coordinates on the low-resolution image; wherein the nearest position is the position of the nearest pixel on the low-resolution image to the high-resolution projection coordinates; determining a predicted high-resolution image according to the nearest position and the local feature of the first pixel feature.

3. The method of claim 2, wherein, The method comprises: acquiring a plurality of adjacent pixel features adjacent to the nearest position, and performing weighted fusion on the plurality of adjacent pixel features adjacent to the nearest position to obtain a weighted pixel feature; determining a predicted high-resolution image according to the weighted pixel feature and the local feature of the first pixel feature.

4. The method of claim 3, wherein, The method comprises: acquiring a plurality of adjacent pixel features adjacent to the nearest position, and performing weighted fusion on the plurality of adjacent pixel features adjacent to the nearest position to obtain a weighted pixel feature; determining a predicted high-resolution image according to the weighted pixel feature and the local feature of the first pixel feature.

5. The method of claim 4, wherein, The method comprises: acquiring a plurality of adjacent pixel features adjacent to the nearest position, and performing weighted fusion on the plurality of adjacent pixel features adjacent to the nearest position to obtain a weighted pixel feature; determining a predicted high-resolution image according to the weighted pixel feature and the local feature of the first pixel feature. The method comprises: determining an intermediate pixel value corresponding to a high-resolution pixel coordinate according to the adjacent pixel coordinates corresponding to each adjacent pixel feature, the high-resolution pixel coordinate, and the weighted pixel feature; determining a pixel value corresponding to the high-resolution pixel coordinate according to the intermediate pixel value and the local feature of the first pixel feature; The prediction high-resolution image is determined according to pixel values corresponding to the plurality of high-resolution pixel coordinates.

6. A super-resolution method characterized by, The method comprises: inputting a low-resolution image into a super-resolution model; wherein the super-resolution model is obtained by training the super-resolution model training method in any one of claims 1-5; obtaining a high-resolution image corresponding to the low-resolution image according to the super-resolution model.

7. An apparatus for training a super-resolution model, comprising: The device comprises: a low-resolution image acquisition module configured to acquire a low-resolution image and input the low-resolution image into a to-be-trained model; wherein the low-resolution image corresponds to a high-resolution image label; the to-be-trained model is a neural network model used for performing a super-resolution reconstruction task; an image pixel feature acquisition module configured to perform feature extraction on the low-resolution image to obtain low-resolution image pixel features; wherein the low-resolution image pixel features include first pixel features corresponding to each pixel; a second pixel feature acquisition module configured to acquire a plurality of second pixel features within a preset range of the first pixel feature; a local feature determination module configured to acquire similarities between the first pixel feature and each of the second pixel features, acquire a plurality of target second pixel features with a similarity greater than a preset threshold, and determine a local feature of the first pixel feature according to the plurality of target second pixel features and the first pixel feature; a model training module configured to determine a prediction high-resolution image according to the local feature of the first pixel feature, update neural network parameters of the to-be-trained model through the prediction high-resolution image and the high-resolution image label, and obtain a super-resolution model.

8. An apparatus for super-resolution, the apparatus comprising: The device comprises: a low-resolution image input module configured to input a low-resolution image into a super-resolution model; wherein the super-resolution model is obtained by training the super-resolution model training method in any one of claims 1-5; a super-resolution reconstruction module configured to obtain a high-resolution image corresponding to the low-resolution image according to the super-resolution model.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the method of any one of claims 1-6.

10. An electronic device, comprising: comprises: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.

Images