Image processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: adjusting the size of an initial image to obtain a first image matrix with a preset pixel size, the initial image being an image with a first resolution; performing serialization processing on the first image matrix by adopting a Hilbert curve to obtain a one-dimensional vector; performing feature extraction on the one-dimensional vector to obtain an image feature vector; and performing image restoration processing on the image feature vector to obtain a target image, the target image being an image with a second resolution, and the first resolution being lower than the second resolution. According to the technical scheme provided by the invention, the space structure information of the image can be effectively reserved, so that the feature extraction capability of the Transform model is enhanced, and the image quality of the target image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] The Transformer model is a novel neural network architecture that has achieved breakthroughs in natural language processing thanks to its unique attention mechanism. This architecture effectively captures global contextual information through attention and builds long-range dependencies, enabling deep extraction of text semantic features. Furthermore, its parallel computing capabilities significantly improve training efficiency. Inspired by this, the application of the Transformer in image processing is increasingly being explored.

[0003] Image data is a matrix, and native Transformers process one-dimensional data. Existing methods often simply convert image matrices into one-dimensional vectors, for example, by directly arranging them in rows and connecting them end to end or in a snake-like manner. This approach has the following drawbacks: adjacent pixels in the image with highly correlated information may become misaligned as the image resolution increases. When the image resolution changes, the relative positions of the pixels in the one-dimensional vector fluctuate dramatically, resulting in the loss of spatial structural information, which in turn affects the model's effective extraction of image features. Therefore, it is urgent to design a processing method that can effectively preserve the spatial structural information of images to fully utilize the performance advantages of Transformers in visual tasks. Summary of the Invention

[0004] The present application provides an image processing method, device, electronic device and storage medium, which can effectively retain the spatial structure information of the image, enhance the feature extraction capability of the Transformer model, and thus improve the image quality of the target image.

[0005] In a first aspect, the present application provides an image processing method, the method comprising:

[0006] Resizing an initial image to obtain a first image matrix of a preset pixel size, wherein the initial image is an image with a first resolution;

[0007] Performing serialization processing on the first image matrix using a Hilbert curve to obtain a one-dimensional vector;

[0008] Performing feature extraction on the one-dimensional vector to obtain an image feature vector;

[0009] Performing image restoration processing on the image feature vector to obtain a target image, where the target image is an image with a second resolution, and the first resolution is lower than the second resolution.

[0010] Furthermore, the first image matrix is ​​serialized using the Hilbert curve to obtain a one-dimensional vector, including: sequentially connecting pixels in the first image matrix according to the first pixel matrix to obtain M first-order Hilbert curves; rotating the first-order Hilbert curves based on the second pixel matrix and sequentially connecting them to obtain N second-order Hilbert curves; repeating the step of rotating the first-order Hilbert curves based on the second pixel matrix and sequentially connecting them six times to obtain L eighth-order Hilbert curves; arranging the eighth-order Hilbert curves in curve order to obtain the one-dimensional vector; wherein, the first pixel matrix is ​​smaller than the second pixel matrix, M, N and L are positive integers, and M is greater than N, and N is greater than L.

[0011] Furthermore, the rotating the first-order Hilbert curves based on the second pixel matrix and then sequentially connecting them to obtain N second-order Hilbert curves includes: dividing the M first-order Hilbert curves into N blocks according to the second pixel matrix, where one block includes four first-order Hilbert curves; for each block, rotating the first-order Hilbert curves and then sequentially connecting them to obtain the second-order Hilbert curve corresponding to each block.

[0012] Furthermore, the feature extraction of the one-dimensional vector to obtain the image feature vector includes: performing position encoding on the one-dimensional vector to obtain a coding vector; performing feature extraction on the coding vector based on a feature extraction model to obtain a first feature, the feature extraction model is composed of a plurality of Transformer Block cascades, each TransformerBlock includes a multi-head self-attention sublayer and a feedforward neural network sublayer; performing a residual connection between the first feature and the coding vector to obtain an enhanced feature; and performing layer normalization on the enhanced feature to obtain the image feature vector.

[0013] Furthermore, the image restoration processing of the image feature vector to obtain the target image includes: using the Hilbert curve sequence to restore the image feature vector to a second image matrix of the preset pixel size; and resizing the second image matrix through a bicubic interpolation algorithm to obtain a target image of the same size as the initial image.

[0014] Furthermore, the step of resizing the initial image to obtain a first image matrix of a preset pixel size includes: resizing the initial image by using a bicubic interpolation algorithm to obtain an image matrix of a preset pixel size.

[0015] Furthermore, if the pixel size of the initial image exceeds a preset threshold, the initial image is divided into multiple overlapping or non-overlapping sub-image blocks along the horizontal and / or vertical direction; each of the sub-image blocks is resized to obtain a first image matrix of the corresponding preset pixel size.

[0016] In a second aspect, the present application provides an image processing device, the device comprising:

[0017] A first processing module, configured to resize an initial image to obtain a first image matrix of a preset pixel size, wherein the initial image is an image with a first resolution;

[0018] A second processing module is used to perform serialization processing on the first image matrix using a Hilbert curve to obtain a one-dimensional vector;

[0019] A feature extraction module, configured to extract features from the one-dimensional vector to obtain an image feature vector;

[0020] An image restoration module is used to perform image restoration processing on the image feature vector to obtain a target image, where the target image is an image with a second resolution, and the first resolution is lower than the second resolution.

[0021] In a third aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image processing method described in any embodiment of the present application.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method described in any embodiment of the present application when executed.

[0023] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the image processing method described in any embodiment of the present application.

[0024] In order to solve the defects of the prior art in the background technology, the embodiment of the present application provides an image processing method, and the execution of this method can bring the following beneficial effects: the present application uses the Hilbert curve to serialize the first image matrix, and when converting the two-dimensional image into a one-dimensional vector, it can retain the local spatial structure information of the image to the greatest extent, and can effectively alleviate the position instability problem of the traditional serialization method during image scaling. Under the premise of maintaining spatial stability, the Transformer model is used to extract features of the one-dimensional vector. Combined with the spatial structure preserved by the Hilbert curve, the extracted image feature vector can more accurately represent the image content. The present application can enhance the feature extraction capability of the Transformer model, improve the image quality of the target image, and give full play to the performance advantages of the Transformer model in visual tasks.

[0025] It should be noted that the above-mentioned computer instructions may be stored in whole or in part on a computer-readable storage medium. The computer-readable storage medium may be packaged together with the processor of the image processing device or separately from the processor of the image processing device, and this application does not limit this.

[0026] The description of the second, third and fourth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third and fourth aspects can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

[0028] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0031] Figure 2A schematic diagram of the model structure of a feature extraction model provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of generating a Hilbert curve provided in an embodiment of the present application;

[0033] Figure 4 A schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0034] Figure 5 This is a block diagram of an electronic device used to implement an image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0036] It should be noted that the terms "first," "second," "target," and "original" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0037] Figure 1 This is a flow chart of an image processing method provided in an embodiment of the present application. This embodiment is applicable to scenarios where an unclear initial image is processed to obtain a super-resolution target image. The image processing method provided in this embodiment can be performed by an image processing device provided in an embodiment of the present application. This device can be implemented in software and / or hardware and integrated into the electronic device that performs the method.

[0038] See also Figure 1 The method of this embodiment includes but is not limited to the following steps:

[0039] S110 , resizing the initial image to obtain a first image matrix of a preset pixel size.

[0040] The initial image is an image with a first resolution, which is not clear and has a low value. The preset pixel size can be determined based on subsequent task requirements and processing system limitations. In deep learning image recognition tasks, different network models have specific requirements for input image size. If the processing device has limited memory, a smaller preset size may be selected to reduce computing resource consumption and avoid memory overflow issues. Optionally, the preset pixel size can be a 256×256 or 128×128 square matrix.

[0041] In the embodiment of the present application, an image scaling algorithm is used to adjust the number of pixels in width and height of the initial image, converting it to a required standard size (i.e., a preset pixel size) to form a first image matrix. Optionally, the scaling algorithm includes at least one of the following: nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, etc. Different algorithms have their own advantages and disadvantages in terms of computational complexity and image quality, and the appropriate algorithm can be selected based on the actual application.

[0042] In one embodiment, when implemented through programming, functions in image processing libraries such as OpenCV can be used to perform the resizing operation. The BiCubic algorithm built into OpenCV is used to convert the initial image into a first image matrix of 256×256 or 128×128 size. Furthermore, to ensure image quality after resizing, pre-processing operations such as color space conversion and sharpening enhancement can also be performed in conjunction with the image.

[0043] In one embodiment, resizing an initial image to obtain a first image matrix of a preset pixel size includes resizing the initial image using a bicubic interpolation algorithm to obtain an image matrix of the preset pixel size. The bicubic interpolation algorithm can generate a new pixel value by calculating the weighted average of a number (e.g., 16) of neighboring pixels surrounding a target pixel. Compared to bilinear interpolation, bicubic interpolation can more accurately restore image details (e.g., textures and edges) and effectively reduce distortion such as aliasing and blurring that occurs during scaling. It is particularly suitable for scenarios requiring high image quality, such as medical imaging and high-definition photographic image processing.

[0044] In one embodiment, when the aspect ratio of the original image is inconsistent with the preset pixel dimensions, there are generally two approaches to avoid image distortion. One is to scale the image proportionally and then fill the remaining area with black pixels, white pixels, or average pixels. The other is to crop the original image, retaining the central area or key content area, and then scale it back to the preset pixel dimensions.

[0045] Furthermore, for oversized initial images, a block processing strategy can be employed. Specifically, if the pixel size of the initial image exceeds a preset threshold, the initial image is segmented horizontally and / or vertically into multiple overlapping or non-overlapping sub-image blocks. Each sub-image block is resized to obtain a first image matrix of the corresponding preset pixel size. Each first image matrix is ​​then resized before being stitched together.

[0046] S120 , performing serialization processing on the first image matrix using the Hilbert curve to obtain a one-dimensional vector.

[0047] Among them, the Hilbert curve is a space-filling curve that can map pixel points in a two-dimensional space to a one-dimensional space while maintaining spatial proximity as much as possible.

[0048] In an embodiment of the present application, based on the Hilbert space-filling curve algorithm, the elements in the two-dimensional matrix (i.e., the first image matrix) are mapped into one-dimensional vectors in the curve traversal order. This mapping maintains the continuity of elements in adjacent spatial positions in the one-dimensional sequence.

[0049] This embodiment maps the first image matrix to a one-dimensional vector through the Hilbert curve, which can effectively preserve the local spatial structure of the first image matrix and improve the Transformer model's ability to capture spatial correlation. Compared with the simple method of directly connecting end to end or snake-like by row priority or column priority, the Hilbert curve can better preserve the local structural information of the image and better maintain the spatial feature distribution of the image. The use of the Hilbert curve for serialization processing can effectively alleviate the position instability problem of traditional serialization methods during image scaling. When the image resolution changes, the spatial distance between adjacent pixels in the original image in the one-dimensional vector obtained based on the Hilbert curve remains relatively stable, so that subsequent one-dimensional vector-based processing (such as feature extraction) can better utilize the spatial structure information of the image and improve the image processing effect. In addition, the fractal characteristics of the Hilbert curve give it multi-scale expression capabilities, which is suitable for processing image data at different resolutions.

[0050] S130 , performing feature extraction on the one-dimensional vector to obtain an image feature vector.

[0051] In an embodiment of the present application, a one-dimensional vector can be used as the input of the Transformer model, and a multi-head attention mechanism can be used to capture the long-distance dependencies of each position in the one-dimensional vector, thereby mining the semantic, structural, and visual information contained in the data to obtain an image feature vector.

[0052] Specifically, feature extraction is performed on a one-dimensional vector to obtain an image feature vector, including: position encoding the one-dimensional vector to obtain a coding vector; feature extraction is performed on the coding vector based on a feature extraction model to obtain a first feature, the feature extraction model is composed of a plurality of Transformer Block cascades, and each Transformer Block includes a multi-head self-attention sublayer and a feedforward neural network sublayer; the first feature is residually connected with the coding vector to obtain an enhanced feature; and the enhanced feature is layer-normalized to obtain an image feature vector.

[0053] Add position encoding information to each element of a one-dimensional vector. Position encoding includes but is not limited to sine-cosine encoding, learnable position embedding, or rotational position encoding to represent the relative position of the element in the sequence. For example, taking sine-cosine encoding as an example, the encoding of an element at a certain position in a one-dimensional vector can be expressed by the following formula:

[0054]

[0055] Among them, pos represents the absolute position index in the one-dimensional vector, which is similar to the position of a word in a sentence in natural language processing; d model Represents the hidden layer dimension of the Transformer model; i represents the group index of the dimension in the encoding vector, with a value range of 0≤i <d model / 2.

[0056] like Figure 2 The following is a schematic diagram of the model structure of the feature extraction model. Figure 2 (a) in the figure shows that the model consists of four Transformer Blocks, each of which is as follows Figure 2 As shown in (b) of Figure 1, the encoded vector obtained through positional encoding is input into the feature extraction model. After processing through four Transformer Blocks, it is added to the input of the feature extraction model (i.e., the encoded vector) using a residual connection to obtain enhanced features. The enhanced features are then layer-normalized and fed into a multi-layer perceptron (MLP) for dimensionality transformation or classification regression to obtain the image feature vector.

[0057] This embodiment combines residual connections and layer normalization to significantly improve the training stability and convergence speed of deep Transformer networks.

[0058] Optionally, the feature extraction model supports modular design, which allows flexible adjustment of the number of Transformer blocks, the number of attention heads, and the MLP structure to adapt to downstream tasks of varying complexity (such as image classification, object detection, and semantic segmentation).

[0059] S140 , performing image restoration processing on the image feature vector to obtain a target image.

[0060] The target image is an image with a second resolution, which is a super-resolution image. The second resolution has a higher value, and the first resolution is lower than the second resolution.

[0061] Specifically, image restoration processing is performed on the image feature vector to obtain a target image, including: using the Hilbert curve sequence to restore the image feature vector to a second image matrix of a preset pixel size; and resizing the second image matrix through a bicubic interpolation algorithm to obtain a target image of the same size as the initial image.

[0062] Based on a preset pixel size (e.g., 256×256 or 128×128), the corresponding Hilbert curve order is calculated (e.g., an eighth-order curve corresponds to 256×256). Based on the recursive rules for constructing the Hilbert curve (e.g., rotation and endpoint connection logic), the elements in the image feature vector are filled into a two-dimensional matrix along the curve traversal path. For example, starting from the order of the first-order curve, the one-dimensional data is relocated to the corresponding pixel positions using the rotation and splicing rules of higher-order curves to generate a second image matrix. The target output size of the bicubic interpolation is set based on the width and height of the initial image, ensuring that the second image matrix is ​​consistent with the original image size after scaling. The sampling point coordinates are calculated based on the target size. For each coordinate position, a cubic spline function or optimized weighting function is used to calculate the weighted sum of several (e.g., 16) neighboring pixels to generate a new pixel value at the corresponding position, completing image enlargement or reduction. Strategies such as mirror padding and edge duplication are used to process image boundaries to avoid abnormal pixel values ​​caused by insufficient sampling range and ensure the integrity of the output image.

[0063] When restoring a one-dimensional image feature vector to a two-dimensional matrix, the inverse mapping properties of the Hilbert curve are utilized to rearrange the elements in the image feature vector to corresponding coordinate positions according to the curve construction rules, thereby restoring the image's two-dimensional spatial structure. Image restoration processing, performed through the inverse Hilbert curve mapping and bicubic interpolation algorithm, ensures that the target image recovered from the image feature vector not only restores the spatial layout of the original image but also leverages the high-precision properties of bicubic interpolation to reduce detail loss during scaling, improve image reconstruction quality, and achieve high-quality closed-loop processing from image input to output.

[0064] Compared with simple sequential filling (such as row priority or column priority), the image restoration processing through the Hilbert curve in this embodiment can retain the local structural information of the image to the maximum extent, avoid the loss of spatial relationships caused by the misalignment of elements in the one-dimensional sequence, and ensure the visual coherence and semantic integrity of the restored image.

[0065] The technical solution provided by this embodiment is to obtain a first image matrix of a preset pixel size by resizing the initial image; to obtain a one-dimensional vector by serializing the first image matrix using the Hilbert curve; to obtain an image feature vector by extracting features from the one-dimensional vector; and to obtain a target image by performing image restoration on the image feature vector. This application uses the Hilbert curve to serialize the first image matrix. When converting a two-dimensional image into a one-dimensional vector, it can retain the local spatial structure information of the image to the greatest extent, and can effectively alleviate the position instability problem of the traditional serialization method during image scaling. Under the premise of maintaining spatial stability, the Transformer model is used to extract features from the one-dimensional vector. Combined with the spatial structure preserved by the Hilbert curve, the extracted image feature vector can more accurately represent the image content. This application can enhance the feature extraction capability of the Transformer model, improve the image quality of the target image, and give full play to the performance advantages of the Transformer model in visual tasks.

[0066] In an optional embodiment, the specific execution process of the above-mentioned step S120 (i.e., serializing the first image matrix using the Hilbert curve to obtain a one-dimensional vector) can be: sequentially connecting the pixels in the first image matrix according to the first pixel matrix (such as 2×2) to obtain M first-order Hilbert curves; rotating the first-order Hilbert curve based on the second pixel matrix (such as 4×4) and sequentially connecting them to obtain N second-order Hilbert curves; repeating the step of rotating the first-order Hilbert curve based on the second pixel matrix and then sequentially connecting them six times to obtain L eighth-order Hilbert curves; arranging the eighth-order Hilbert curves according to the curve order to obtain a one-dimensional vector; wherein, the first pixel matrix is ​​smaller than the second pixel matrix, M, N and L are positive integers, and M is greater than N, and N is greater than L.

[0067] like Figure 3 The following is a schematic diagram of the Hilbert curve generation. First, follow Figure 3 The 2×2 pixel matrix in (a) is connected in the order of ACDB, which is the first-order Hilbert curve. Figure 3 (b) is a 4×4 pixel matrix, where A1, B1, C1, and D1 are all first-order Hilbert curves. At this time, A1 is rotated 90°, B1 is rotated -90°, and the adjacent endpoints of the four blocks are connected, as shown in Figure 3 The second-order Hilbert curve is shown in (c). Similarly, four second-order Hilbert curves can be connected after rotation to form a third-order Hilbert curve, as shown in Figure 3 Similarly, the desired eighth-order Hilbert curve of a 256×256 or 128×128 pixel matrix can be obtained.

[0068] The method includes rotating the first-order Hilbert curve based on the second pixel matrix and sequentially connecting them to obtain N second-order Hilbert curves, including: dividing the M first-order Hilbert curves into N blocks according to the second pixel matrix, where one block includes four first-order Hilbert curves; and for each block, rotating the first-order Hilbert curve and sequentially connecting them to obtain the second-order Hilbert curve corresponding to each block.

[0069] The step of rotating the first-order Hilbert curve based on the second pixel matrix and then sequentially connecting them is repeated six times to obtain L eighth-order Hilbert curves, including:

[0070] In the first repetition, the N second-order Hilbert curves are divided into O blocks according to the third pixel matrix (e.g., 8×8), with each block containing four second-order Hilbert curves. For each block, the second-order Hilbert curves are rotated and then sequentially connected to obtain the third-order Hilbert curve corresponding to each block, thereby obtaining O third-order Hilbert curves. Where O is a positive integer and N is greater than O.

[0071] Repeat the process a second time, dividing the O second-order Hilbert curves into P blocks according to a fourth pixel matrix (e.g., 16×16), with each block containing four third-order Hilbert curves. For each block, the third-order Hilbert curves are rotated and then sequentially connected to obtain a fourth-order Hilbert curve corresponding to each block, thereby obtaining P fourth-order Hilbert curves. P is a positive integer, and O is greater than P.

[0072] Repeat the process for the third time, dividing the P second-order Hilbert curves into Q blocks according to the fifth pixel matrix (e.g., 32×32), where each block includes four fourth-order Hilbert curves. For each block, the fourth-order Hilbert curves are rotated and then sequentially connected to obtain a fifth-order Hilbert curve corresponding to each block, thereby obtaining Q fifth-order Hilbert curves. Here, Q is a positive integer, and P is greater than Q.

[0073] Repeat the fourth time, dividing the Q second-order Hilbert curves into R blocks according to the fifth pixel matrix (e.g., 64×64), where each block includes four fifth-order Hilbert curves. For each block, the fifth-order Hilbert curves are rotated and then sequentially connected to obtain a sixth-order Hilbert curve corresponding to each block, thereby obtaining R sixth-order Hilbert curves. R is a positive integer, and Q is greater than R.

[0074] The fifth repetition is to divide the R second-order Hilbert curves into S blocks according to the sixth pixel matrix (e.g., 128×128), with each block containing four sixth-order Hilbert curves. For each block, the sixth-order Hilbert curves are rotated and then sequentially connected to obtain a seventh-order Hilbert curve corresponding to each block, thereby obtaining S seventh-order Hilbert curves. Where S is a positive integer, and R is greater than S.

[0075] The sixth repetition is to divide the S second-order Hilbert curves into L blocks according to the seventh pixel matrix (256×256), with each block containing four seventh-order Hilbert curves. For each block, the seventh-order Hilbert curves are rotated and then sequentially connected to obtain the eighth-order Hilbert curve corresponding to each block, thereby obtaining L eighth-order Hilbert curves. Where L is a positive integer and S is greater than L.

[0076] Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the apparatus 400 may include:

[0077] A first processing module 410 is configured to resize an initial image to obtain a first image matrix of a preset pixel size, wherein the initial image is an image with a first resolution;

[0078] A second processing module 420 is configured to perform serialization processing on the first image matrix using a Hilbert curve to obtain a one-dimensional vector;

[0079] A feature extraction module 430 is configured to extract features from the one-dimensional vector to obtain an image feature vector;

[0080] The image restoration module 440 is configured to perform image restoration processing on the image feature vector to obtain a target image, where the target image is an image with a second resolution, and the first resolution is lower than the second resolution.

[0081] Furthermore, the above-mentioned second processing module 420 can be specifically used to: sequentially connect the pixels in the first image matrix according to the first pixel matrix to obtain M first-order Hilbert curves; rotate the first-order Hilbert curves based on the second pixel matrix and sequentially connect them to obtain N second-order Hilbert curves; repeat the step of rotating the first-order Hilbert curves based on the second pixel matrix and sequentially connecting them six times to obtain L eighth-order Hilbert curves; arrange the eighth-order Hilbert curves in curve order to obtain the one-dimensional vector; wherein, the first pixel matrix is ​​smaller than the second pixel matrix, M, N and L are positive integers, and M is greater than N, and N is greater than L.

[0082] Furthermore, the above-mentioned second processing module 420 can also be specifically used to: divide the M first-order Hilbert curves into N blocks according to the second pixel matrix, and one block includes four first-order Hilbert curves; for each block, the first-order Hilbert curve is rotated and then sequentially connected to obtain the second-order Hilbert curve corresponding to each block.

[0083] Furthermore, the above-mentioned feature extraction module 430 can be specifically used to: perform position encoding on the one-dimensional vector to obtain a coding vector; perform feature extraction on the coding vector based on a feature extraction model to obtain a first feature, the feature extraction model is composed of a plurality of Transformer Block cascades, and each Transformer Block includes a multi-head self-attention sublayer and a feedforward neural network sublayer; perform a residual connection on the first feature and the coding vector to obtain an enhanced feature; perform layer normalization on the enhanced feature to obtain the image feature vector.

[0084] Furthermore, the above-mentioned image restoration module 440 can be specifically used to: use the Hilbert curve sequence to restore the image feature vector to the second image matrix of the preset pixel size; resize the second image matrix through the bicubic interpolation algorithm to obtain a target image of the same size as the initial image.

[0085] Furthermore, the first processing module 410 may be specifically configured to resize the initial image using a bicubic interpolation algorithm to obtain an image matrix of a preset pixel size.

[0086] Furthermore, the above-mentioned first processing module 410 can also be specifically used to: before resizing the initial image to obtain the first image matrix of the preset pixel size, if the pixel size of the initial image exceeds a preset threshold, divide the initial image into multiple overlapping or non-overlapping sub-image blocks along the horizontal and / or vertical direction; and resize each of the sub-image blocks separately to obtain the first image matrix of the corresponding preset pixel size.

[0087] The image processing device provided in this embodiment can be applied to the image processing method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0088] Figure 51 is a block diagram of an electronic device for implementing an image processing method according to an embodiment of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0089] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0090] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0091] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the image processing method.

[0092] In some embodiments, the image processing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the image processing method in any other suitable manner (e.g., via firmware).

[0093] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] Computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0095] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0098] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0099] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. For example, those skilled in the art can use the various forms of processes shown above, reorder, add, or delete steps; and can perform the steps described in the present application in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present application can be achieved, and this document does not limit them here.

[0100] The above specific embodiments do not limit the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that: The method comprises: Resizing an initial image to obtain a first image matrix of a preset pixel size, wherein the initial image is an image with a first resolution; Performing serialization processing on the first image matrix using a Hilbert curve to obtain a one-dimensional vector; Performing feature extraction on the one-dimensional vector to obtain an image feature vector; Performing image restoration processing on the image feature vector to obtain a target image, where the target image is an image with a second resolution, and the first resolution is lower than the second resolution.

2. The image processing method according to claim 1, wherein: The serialization process of the first image matrix using the Hilbert curve to obtain a one-dimensional vector includes: sequentially connecting pixels in the first image matrix according to the first pixel matrix to obtain M first-order Hilbert curves; The first-order Hilbert curves are rotated based on the second pixel matrix and then sequentially connected to obtain N second-order Hilbert curves; Repeating the step of rotating the first-order Hilbert curve based on the second pixel matrix and then sequentially connecting the first-order Hilbert curves six times to obtain L eighth-order Hilbert curves; Arranging the eighth-order Hilbert curve in curve order to obtain the one-dimensional vector; The first pixel matrix is ​​smaller than the second pixel matrix, M, N and L are positive integers, and M is greater than N, and N is greater than L.

3. The image processing method according to claim 2, wherein: The first-order Hilbert curves are rotated based on the second pixel matrix and then sequentially connected to obtain N second-order Hilbert curves, including: Dividing the M first-order Hilbert curves into N blocks according to the second pixel matrix, where one block includes four first-order Hilbert curves; For each block, the first-order Hilbert curve is rotated and then sequentially connected to obtain the second-order Hilbert curve corresponding to each block.

4. The image processing method according to claim 1, wherein: The step of extracting features from the one-dimensional vector to obtain an image feature vector includes: Performing position encoding on the one-dimensional vector to obtain an encoding vector; Performing feature extraction on the encoding vector based on a feature extraction model to obtain a first feature, wherein the feature extraction model is composed of a plurality of Transformer Block cascades, each Transformer Block including a multi-head self-attention sublayer and a feedforward neural network sublayer; Performing a residual connection between the first feature and the encoding vector to obtain an enhanced feature; Perform layer normalization processing on the enhanced features to obtain the image feature vector.

5. The image processing method according to claim 1, wherein: The performing image restoration processing on the image feature vector to obtain a target image includes: Reducing the image feature vector to a second image matrix of the preset pixel size using a Hilbert curve sequence; The second image matrix is ​​resized using a bicubic interpolation algorithm to obtain a target image with the same size as the initial image.

6. The image processing method according to claim 1, wherein: The step of resizing the initial image to obtain a first image matrix of a preset pixel size includes: The initial image is resized using a bicubic interpolation algorithm to obtain an image matrix of a preset pixel size.

7. The image processing method according to claim 1, wherein: The step of resizing the initial image to obtain a first image matrix of a preset pixel size includes: If the pixel size of the initial image exceeds a preset threshold, dividing the initial image into a plurality of overlapping or non-overlapping sub-image blocks along the horizontal and / or vertical directions; The size of each sub-image block is adjusted respectively to obtain a first image matrix of a corresponding preset pixel size.

8. An image processing device, characterized in that: The device comprises: A first processing module, configured to resize an initial image to obtain a first image matrix of a preset pixel size, wherein the initial image is an image with a first resolution; A second processing module is used to perform serialization processing on the first image matrix using a Hilbert curve to obtain a one-dimensional vector; A feature extraction module, configured to extract features from the one-dimensional vector to obtain an image feature vector; An image restoration module is used to perform image restoration processing on the image feature vector to obtain a target image, where the target image is an image with a second resolution, and the first resolution is lower than the second resolution.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method according to any one of claims 1 to 7 when executed.

Images