Image data enhancement method and device, equipment, storage medium and program product
By extracting and fusing feature images with high similarity and original images in the convolutional neural network, the problem of low image data enhancement quality in the prior art is solved, and higher quality image data enhancement is achieved.
Patent Information
- Application Number
- CN202510621892.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2025-08-15
AI Technical Summary
In the prior art, the method of generating new images for image data enhancement through random transformation results in low quality of newly generated images, affecting the quality of image data enhancement.
By inputting the original image into the pre-trained convolutional neural network, the feature image output from the convolutional layer before the first pooling layer is obtained, and according to the difference between the feature image and the original image, the feature image with high similarity is selected to fuse it with the original image, and the target fusion image is generated for data enhancement.
The quality of newly generated images is improved, thereby improving the effect of image data enhancement.
Smart Images
Figure CN120495841A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image data enhancement method, apparatus, device, storage medium, and program product. Background Art
[0002] Deep learning models are widely used in data processing in banking business. In order to solve the problem of insufficient image training data during the training process of deep learning models, image data augmentation technology is introduced to expand the image training data.
[0003] In related technologies, new images are generated for image data enhancement by performing random changes on existing images, such as rotating, flipping, or cropping the images.
[0004] The new image generated by random transformations easily changes the characteristic distribution of pixels in the original image, resulting in the weakening of the main features of the new image and low quality of the newly generated image, which in turn affects the quality of image data enhancement. Therefore, how to improve the quality of image data enhancement has become a technical problem that needs to be solved urgently. Summary of the Invention
[0005] The embodiments of the present application provide an image data enhancement method, apparatus, device, storage medium, and program product to achieve the technical effect of improving the quality of image data enhancement.
[0006] In a first aspect, an embodiment of the present application provides an image data enhancement method, comprising:
[0007] Get the original image;
[0008] Inputting the original image into a pre-trained convolutional neural network to obtain first feature images output by at least one convolutional layer before the first pooling layer; wherein the resolution of each first feature image is equal to the resolution of the original image;
[0009] determining at least one second feature image from at least one of the first feature images according to a difference between each of the first feature images and the original image;
[0010] For each second feature image, the second feature image is fused with the original image to obtain a target fused image of the original image. The target fused image is used to perform image data enhancement on the original image.
[0011] In a second aspect, an embodiment of the present application provides an image data enhancement device, comprising:
[0012] An acquisition module is used to acquire the original image;
[0013] a feature extraction module, configured to input the original image into a pre-trained convolutional neural network and obtain first feature images output by at least one convolutional layer before the first pooling layer; wherein the resolution of each first feature image is equal to the resolution of the original image;
[0014] a difference comparison module, configured to determine at least one second feature image from at least one of the first feature images based on a difference between each of the first feature images and the original image;
[0015] The fusion module fuses each second feature image with the original image to obtain a target fused image of the original image, wherein the target fused image is used to perform image data enhancement on the original image.
[0016] In a third aspect, an embodiment of the present application provides an image data enhancement device, including: a memory, a processor;
[0017] The memory stores computer-executable instructions;
[0018] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.
[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.
[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0022] Figure 1 A schematic diagram of the related technology provided by this application using a rotation transformation method to generate a new image;
[0023] Figure 2 A flowchart of the image data enhancement method provided in an embodiment of the present application;
[0024] Figure 3 A schematic diagram of a process for calculating the Hamming distance between a first hash feature and a second hash feature provided in an embodiment of the present application;
[0025] Figure 4 A schematic diagram of the process of fusing a feature image with a single-channel original image provided in an embodiment of the present application;
[0026] Figure 5 A schematic diagram of the process of fusing a feature image with a multi-channel original image provided in an embodiment of the present application;
[0027] Figure 6 A schematic diagram of the structure of an image data enhancement device provided in an embodiment of the present application;
[0028] Figure 7 A schematic diagram of the structure of the image data enhancement device provided in an embodiment of the present application.
[0029] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION
[0030] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
[0031] Deep learning models are widely used in data processing for banking applications. The results of deep learning data processing are influenced by many factors, one of which is the amount of training data used to train the deep learning model. Generally, the larger the amount of training data, the more accurate the data processing results of the deep learning model trained with it. However, for some banking applications, obtaining large quantities of training data is difficult. Therefore, image augmentation technology can be used to expand the training data.
[0032] In related technologies, a new image is generated for image data enhancement by performing random transformations on an existing image, for example, rotating, flipping, or cropping the image. Figure 1 The related technology provided by this application uses a rotation transformation method to generate a new image, such as Figure 1As shown in the figure, in the rectangular coordinate system formed by the x and y axes, the rectangular box is the valid area of the original image, θ is the angle of image rotation, (x1, y1) is the pixel in the original image, and (x2, y2) is the pixel obtained by rotating the pixel (x1, y1) by angle θ. Since the valid area of the image is within the rectangular box, after the image is rotated, the pixel (x2, y2) falls outside the rectangular box, resulting in the loss of the feature of the pixel (x1, y1) in the original image. The generated new image changes the feature distribution of the pixels in the original image, which may weaken the main features of the image in the new image, thereby affecting the quality of the newly generated image. The new image is used for image data enhancement, resulting in low quality of image data enhancement.
[0033] Based on the above analysis, it can be seen that in the prior art, the method of generating new images through random transformation for image data enhancement has a technical problem that the quality of the newly generated images is low, thereby resulting in low quality of the image data enhancement.
[0034] The image data enhancement method, device, equipment, storage medium and program product provided in this application are used to solve the above-mentioned technical problems.
[0035] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0036] Figure 2 A flow chart of the image data enhancement method provided in the embodiment of the present application is shown as follows: Figure 2 As shown, the method includes:
[0037] S210: Acquire an original image.
[0038] It should be noted that the image set for data augmentation can be a training image dataset for training a deep learning model, and the obtained original image can be any image in the training image dataset.
[0039] S220. Input the original image into a pre-trained convolutional neural network to obtain first feature images output by at least one convolutional layer before the first pooling layer; wherein the resolution of each first feature image is equal to the resolution of the original image.
[0040] It should be noted that if the original image is a single-channel image, the first feature image processed by the convolution layer corresponds to a single-channel image; if the original image is a multi-channel image, the first feature image processed by the convolution layer corresponds to a multi-channel image.
[0041] The pre-trained convolutional neural network can be a convolutional neural network used to process images, and the image processing algorithm can be determined based on specific banking business. Based on the image processing algorithm, the initial convolutional neural network can be preliminarily trained using the original training image data set to obtain a pre-trained convolutional neural network for image data enhancement. The pre-trained convolutional neural network can be a convolutional neural network with multiple convolutional layers and multiple pooling layers. The convolution layer includes a convolution kernel, which can be used to extract image features such as edges and textures of the input image; the pooling layer retains the most significant image feature responses by downsampling the image output by the previous layer, thereby reducing the amount of computation; the convolution layer before the first pooling layer can use the convolution kernel to process the input image, thereby outputting a feature image with the same resolution as the original image.
[0042] Exemplarily, the first feature image can be obtained by using three shallow convolution layers before the first pooling layer of the convolutional neural network. The three shallow convolution layers are the first shallow convolution layer, the second shallow convolution layer, and the third shallow convolution layer. For an input original image, the first shallow convolution layer, the second shallow convolution layer, and the third shallow convolution layer respectively output the first feature image of the corresponding layer, and the resolution of the first feature image is equal to the resolution of the original image. The matrix size of the convolution kernel used by the first shallow convolution layer, the second shallow convolution layer, and the third shallow convolution layer can be different. This application does not limit the matrix size of the convolution kernel used by the convolution layer.
[0043] S230 : Determine at least one second feature image from at least one first feature image according to the difference between each first feature image and the original image.
[0044] It should be noted that the difference between the first feature image and the original image can be used to characterize the similarity of image features between the first feature image and the original image. Based on the difference between each first feature image and the original image, an image that is more similar to the original image in image features can be determined as the second feature image. The determined second feature image can be used to generate a new image in S204 for image data enhancement.
[0045] S240 : For each second feature image, fuse the second feature image with the original image to obtain a target fused image of the original image, where the target fused image is used to perform image data enhancement on the original image.
[0046] The image data enhancement method provided in the embodiment of the present application generates at least one first feature image by performing image processing on the original image using a convolutional layer before the first pooling layer of a convolutional neural network; determines at least one second feature image having features close to those of the original image based on the difference between the feature image and the original image; fuses each second feature image with the original image, and the obtained target fused image is used for image data enhancement. Since, in a neural network, processing the image using the convolutional layer before the first pooling layer can well extract the edge and texture features of the image, the generated first feature image retains the edge and texture features of the original image; by calculating the difference between the first feature image and the original image, a second feature image having features close to those of the original image is determined for fusion with the original image, and the generated fused image is used for image data enhancement, thereby improving the quality of the newly generated image, thereby improving the quality of the image data enhancement.
[0047] In some specific embodiments, the above S230 includes the following sub-steps:
[0048] S231 . For each first feature image, calculate the similarity between the first feature image and the original image based on a similarity algorithm.
[0049] S232: Determine at least one second feature image according to the similarity between each first feature image and the original image.
[0050] It should be noted that the greater the similarity between the first feature image and the original image, the more detailed features of the original image can be retained by the obtained fused image when the first feature image is used for image fusion in subsequent operations.
[0051] As an example, at least one first feature image can be sorted from large to small according to the similarity between each first feature image and the original image, and at least one first feature image ranked in the top 50% is taken as at least one second feature image for image fusion in subsequent operations.
[0052] As another example, a similarity threshold may be set based on historical experience, and at least one first feature image with a similarity greater than the similarity threshold may be used as at least one second feature image for image fusion in subsequent operations.
[0053] In these embodiments, by calculating the similarity between each first feature image and the original image, at least one second feature image is determined based on the similarity, and the determined at least one second feature image is used for image fusion in subsequent operations, ensuring that the feature image used for fusion retains the image detail features of the original image while being different from the original image, thereby improving the quality of the generated fused image and further improving the quality of image data enhancement.
[0054] In some implementations of these embodiments, the above S231 includes the following sub-steps:
[0055] S2311 . Map the first feature image to a first grayscale image of a preset resolution based on a preset image processing algorithm.
[0056] It should be noted that the preset resolution is smaller than the resolution of the first feature image. By mapping the first feature image to the first grayscale image of the preset resolution, the amount of subsequent image calculations can be reduced, thereby improving the efficiency of the algorithm.
[0057] Exemplarily, the resolution of the first feature image is 90 pixels in width and 80 pixels in height, and the first feature image can be mapped to a first grayscale image with a width of 9 pixels and a height of 8 pixels.
[0058] If the first feature image is a multi-channel image, an image grayscale algorithm may be used to map the multi-channel image to the first grayscale image. This application does not limit the image grayscale algorithm.
[0059] As an example, the pixel values of pixels at the same position and different channels in the first feature image may be averaged, and the average value may be used as the pixel value of the pixel at the corresponding position in the first grayscale image to obtain the first grayscale image.
[0060] As another example, a weighted average of the pixel values of pixels at the same position and different channels in the first feature image may be taken, and the weighted average may be used as the pixel value of the pixels at corresponding positions in the first grayscale image to obtain the first grayscale image.
[0061] S2312: Based on a preset hash algorithm, determine a first hash feature of the first grayscale image according to pixel values of multiple pixels of the first grayscale image.
[0062] S2313: Map the original image to a second grayscale image with a preset resolution based on a preset image processing algorithm.
[0063] It should be noted that the preset image processing algorithms for image processing the first feature image and the original image are the same, thereby ensuring that the first grayscale image and the second grayscale image have the same resolution, and the same image processing algorithm is used to process the images, ensuring the consistency of the processing process of the first grayscale image and the second grayscale image, thereby ensuring the accuracy of the subsequent calculated similarity.
[0064] S2314: Based on a preset hash algorithm, determine a second hash feature of the second grayscale image according to pixel values of multiple pixels of the second grayscale image.
[0065] S2315 , calculating a target Hamming distance between the first hash feature and the second hash feature, and determining a similarity between the first feature image and the original image according to the target Hamming distance.
[0066] In these embodiments, by extracting the respective hash features of the first feature image and the original image, using the hash features to calculate the Hamming distance between the first feature image and the original image, and determining the similarity between the first feature image and the original image based on the Hamming distance, it is ensured that the feature image used for fusion in subsequent operations retains the image detail features of the original image, and the fusion of different feature images with the original image can highlight the different feature points of the original image, thereby enriching the image data set.
[0067] In some implementations of these embodiments, the above S2312 includes the following sub-steps:
[0068] First, multiple pixels of the first grayscale image are traversed row by row. For each row of pixels, the pixel values between any two adjacent pixels are compared from left to right; if the pixel value of the first pixel on the left is greater than the pixel value of the second pixel on the right, then the comparison result between the first pixel and the second pixel is determined to be 1.
[0069] Second, if the pixel value of the first pixel on the left is less than or equal to the pixel value of the second pixel on the right, then the comparison result between the first pixel and the second pixel is determined to be 0.
[0070] Third, based on a preset order, multiple comparison results of pixels in each row of the first feature image are combined to obtain a first hash feature.
[0071] Regarding the preset order in step 3, the multiple comparison results may be combined according to the arrangement order of the pixels corresponding to the comparison results from left to right and from top to bottom in the first feature image.
[0072] For example, for a first grayscale image with a resolution of 9 pixels in width and 8 pixels in height, each row of pixels in the first grayscale image generates 8 comparison results from left to right, and 8 rows of pixels generate 8 times 8, which equals 64 comparison results from top to bottom. The 64 comparison results are combined in order from left to right and from top to bottom to obtain the first hash feature.
[0073] In these embodiments, the pixels in the feature image are traversed, and the pixel values of any two adjacent pixels in each row are sequentially compared to obtain a comparison result for the adjacent pixels. Multiple comparison results are combined to form a hash feature for the grayscale image. The hash feature extracts deeper image features from the grayscale image, thereby ensuring the accuracy of subsequent similarity calculations.
[0074] Figure 3 A schematic diagram of a process for calculating the Hamming distance between the first hash feature and the second hash feature provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, in some implementations of these embodiments, the target Hamming distance between the first hash feature and the second hash feature is obtained based on the following operations:
[0075] S301. Set the initial value S of the Hamming distance to 0.
[0076] S302 , performing a first traversal to traverse multiple comparison results in the first hash feature, and for each first comparison result in the first hash feature, obtaining a second comparison result at a corresponding position in the second hash feature.
[0077] S303: If the first comparison result is not equal to the second comparison result, update S with S+1; otherwise, keep S unchanged.
[0078] S304 : In response to the completion of the first traversal, output an updated S as a target Hamming distance between the first hash feature and the second hash feature.
[0079] In these embodiments, by traversing multiple comparison results in the hash feature, for a first comparison result in the first hash feature and a corresponding second comparison result in the second hash feature, if the first comparison result and the second comparison result are different, the Hamming distance is increased by 1, so that the greater the difference between the first hash feature and the second hash feature, the greater the resulting Hamming distance. Therefore, the resulting Hamming distance can accurately reflect the similarity of image features between the first feature image and the original image.
[0080] Figure 4 The flowchart of the fusion of the feature image and the single-channel original image provided in the embodiment of the present application is as follows: Figure 4 As shown, in some implementations of these embodiments, the above S240 includes the following steps:
[0081] S401: In response to the original image being a single-channel image, perform a second pass, and perform the following fusion operation during the second pass:
[0082] S402 . For each third pixel in the original image, obtain a pixel value of the third pixel and a pixel value of a fourth pixel corresponding to the position of the third pixel in the second feature image.
[0083] S403 : Calculate a first weighted average value of the pixel value of the third pixel and the pixel value of the fourth pixel based on the preset first fusion weight of the original image and the preset second fusion weight of the second feature image.
[0084] S404: Use the first weighted average value as a target pixel value of a pixel corresponding to the third pixel in the target fused image.
[0085] S405 : In response to the second traversal being completed, output a target fused image including a plurality of pixels having target pixel values.
[0086] It should be noted that the preset first fusion weight of the original image and the preset second fusion weight of the second feature image can be fusion weights derived from historical experience. When the original image is a single-channel image, the second feature image and the target fused image are both single-channel images.
[0087] Exemplarily, the first weighted average value of the pixel value of the third pixel and the pixel value of the fourth pixel can be obtained based on the following formula (1):
[0088] m(i,j)=ω1×o(i,j)+ω2×f(i,j) (1);
[0089] Where m(i, j) represents the pixel value of the pixel at position (i, j) of the target fusion image of the single channel; ω1 represents the preset first fusion weight; o(i, j) represents the pixel value of the third pixel at position (i, j) of the original image of the single channel; ω2 represents the preset second fusion weight; f(i, j) represents the pixel value of the fourth pixel at position (i, j) of the second feature image of the single channel.
[0090] In these embodiments, by traversing the pixels of the single-channel original image, weighted fusion is performed on the pixel values of the pixels at corresponding positions of the single-channel original image and the second feature image, and the fused pixel values are used as the pixel values of the pixels at corresponding positions in the fused image, so that the fused image retains the image pixel features of the original image while being different from the original image, so that the fused image can be used for image data enhancement and the quality of image data enhancement is improved.
[0091] Figure 5The flowchart of the fusion of the feature image and the multi-channel original image provided in the embodiment of the present application is as follows: Figure 5 As shown, the above S240 includes the following steps:
[0092] S501: In response to the original image being a multi-channel image, a third pass is performed, in which the following fusion operation is performed:
[0093] S502: For each fifth pixel in the original image, obtain a pixel value of any channel of the fifth pixel and a pixel value of the channel of a sixth pixel corresponding to the position of the fifth pixel in the second feature image.
[0094] S503 : Calculate a second weighted average of the pixel value of the channel of the fifth pixel and the pixel value of the channel of the sixth pixel based on the preset third fusion weight of the original image and the preset fourth fusion weight of the second feature image.
[0095] S504: Use the second weighted average value as the target pixel value of the channel of the pixel corresponding to the fifth pixel in the target fused image.
[0096] S505 : In response to the end of the third traversal, output a target fused image including a plurality of pixels having target pixel values in each channel.
[0097] It should be noted that the preset third fusion weight of the original image and the preset fourth fusion weight of the second feature image can be fusion weights derived from historical experience. When the original image is a multi-channel image, the second feature image and the target fused image are both multi-channel images.
[0098] Exemplarily, the second weighted average value of the pixel value of the fifth pixel and the pixel value of the sixth pixel can be obtained based on the following formula (2):
[0099] g(i,j)=ω3×p(i,j)+ω4×q(i,j) (2);
[0100] Wherein, g(i,j) represents the pixel value of the pixel of a certain channel at the position (i,j) of the multi-channel target fusion image; ω3 represents the preset third fusion weight; p(i,j) represents the pixel value of the fifth pixel of the corresponding channel at the position (i,j) of the multi-channel original image; ω4 represents the preset fourth fusion weight; q(i,j) represents the pixel value of the sixth pixel of the corresponding channel at the position (i,j) of the second feature image of the multi-channel.
[0101] In these embodiments, by traversing the pixels of each channel at each position of the multi-channel original image, the pixel values of the pixels of the corresponding channels at the corresponding positions of the original image and the second feature image are weightedly fused, and the fused pixel values are used as the pixel values of the pixels at the corresponding positions in the fused image, so that the fused image retains the image pixel features of the original image while being different from the original image, so that the fused image can be used for image data enhancement and the quality of image data enhancement is improved.
[0102] Figure 6 This is a schematic diagram of the structure of the image data enhancement device provided in the embodiment of the present application, as shown in FIG. Figure 6 As shown, the image data enhancement device 60 provided in this embodiment includes:
[0103] An acquisition module 601 is used to acquire an original image;
[0104] A feature extraction module 602 is configured to input the original image into a pre-trained convolutional neural network and obtain first feature images output by at least one convolutional layer before the first pooling layer; wherein the resolution of each first feature image is equal to the resolution of the original image;
[0105] a difference comparison module 603, configured to determine at least one second feature image from at least one first feature image according to the difference between each first feature image and the original image;
[0106] The fusion module 604 is configured to fuse each second feature image with the original image to obtain a target fused image of the original image. The target fused image is used to perform image data enhancement on the original image.
[0107] In one possible implementation, the difference comparison module 603 is further configured to calculate, for each first feature image, a similarity between the first feature image and the original image based on a similarity algorithm;
[0108] At least one second feature image is determined according to the similarity between each first feature image and the original image.
[0109] In one possible implementation, the difference comparison module 603 is further configured to map the first feature image to a first grayscale image of a preset resolution based on a preset image processing algorithm;
[0110] Determining, based on a preset hash algorithm and according to pixel values of a plurality of pixels of the first grayscale image, a first hash feature of the first grayscale image;
[0111] Based on a preset image processing algorithm, mapping the original image to a second grayscale image of a preset resolution;
[0112] Determining, based on a preset hash algorithm and according to pixel values of a plurality of pixels of the second grayscale image, a second hash feature of the second grayscale image;
[0113] A target Hamming distance between the first hash feature and the second hash feature is calculated, and a similarity between the first feature image and the original image is determined according to the target Hamming distance.
[0114] In one possible implementation, the difference comparison module 603 is further configured to traverse the plurality of pixels of the first grayscale image row by row, and for each row of pixels, compare the pixel values of any two adjacent pixels from left to right; if the pixel value of the first pixel on the left is greater than the pixel value of the second pixel on the right, then determine that the comparison result between the first pixel and the second pixel is 1;
[0115] If the pixel value of the first pixel on the left is less than or equal to the pixel value of the second pixel on the right, then determining that the comparison result between the first pixel and the second pixel is 0;
[0116] Based on a preset order, multiple comparison results of pixels in each row of the first feature image are combined to obtain a first hash feature.
[0117] In one possible implementation, the difference comparison module 603 is further configured to:
[0118] Set the initial value S of the Hamming distance to 0;
[0119] Perform a first traversal to traverse multiple comparison results in the first hash feature, and for each first comparison result in the first hash feature, obtain a second comparison result at a corresponding position in the second hash feature;
[0120] If the first comparison result is not equal to the second comparison result, update S with S+1, otherwise keep S unchanged;
[0121] In response to the end of the first traversal, the updated S is output as the target Hamming distance between the first hash feature and the second hash feature.
[0122] In one possible implementation, the fusion module 604 is further configured to:
[0123] In response to the original image being a single-channel image, a second pass is performed, during which the following fusion operation is performed:
[0124] For each third pixel in the original image, obtaining a pixel value of the third pixel and a pixel value of a fourth pixel corresponding to the position of the third pixel in the second feature image;
[0125] Calculating a first weighted average of a pixel value of the third pixel and a pixel value of the fourth pixel based on a preset first fusion weight of the original image and a preset second fusion weight of the second feature image;
[0126] using the first weighted average as a target pixel value of a pixel corresponding to the third pixel in the target fused image;
[0127] In response to the second traversal ending, a target fused image including a plurality of pixels having target pixel values is output.
[0128] In one possible implementation, the fusion module 604 is further configured to:
[0129] In response to the original image being a multi-channel image, a third pass is performed, in which the following fusion operation is performed:
[0130] For each fifth pixel in the original image, obtain a pixel value of any channel of the fifth pixel and a pixel value of the channel of a sixth pixel corresponding to the position of the fifth pixel in the second feature image;
[0131] Calculating a second weighted average of the pixel value of the channel of the fifth pixel and the pixel value of the channel of the sixth pixel based on a preset third fusion weight of the original image and a preset fourth fusion weight of the second feature image;
[0132] Using the second weighted average as the target pixel value of the channel of the pixel corresponding to the fifth pixel in the target fused image;
[0133] In response to the end of the third traversal, a target fused image including a plurality of pixels having target pixel values in each channel is output.
[0134] The image data enhancement device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.
[0135] Figure 7 This is a schematic diagram of the structure of the image data enhancement device provided by this application. Figure 7 As shown, the image data enhancement device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702, and the communication component 703 are connected via a bus.
[0136] During the specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that the at least one processor 701 performs the above method.
[0137] The specific implementation process of the processor 701 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.
[0138] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules in the processor.
[0139] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.
[0140] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.
[0141] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0142] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0143] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0144] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0145] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.
[0146] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0147] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0148] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.
[0149] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0150] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.
Claims
1. A method for image data enhancement, characterized in that: include: Get the original image; Inputting the original image into a pre-trained convolutional neural network to obtain first feature images output by at least one convolutional layer before the first pooling layer; wherein the resolution of each first feature image is equal to the resolution of the original image; determining at least one second feature image from at least one of the first feature images according to a difference between each of the first feature images and the original image; For each second feature image, the second feature image is fused with the original image to obtain a target fused image of the original image. The target fused image is used to perform image data enhancement on the original image.
2. The method according to claim 1, characterized in that The determining at least one second feature image from at least one of the first feature images according to the difference between each of the first feature images and the original image comprises: For each first feature image, calculating the similarity between the first feature image and the original image based on a similarity algorithm; At least one second feature image is determined according to the similarity between each first feature image and the original image.
3. The method according to claim 2, characterized in that The step of calculating, for each first feature image, the similarity between the first feature image and the original image based on a similarity algorithm includes: Based on a preset image processing algorithm, mapping the first feature image to a first grayscale image with a preset resolution; determining, based on a preset hash algorithm and according to pixel values of a plurality of pixels of the first grayscale image, a first hash feature of the first grayscale image; Mapping the original image to a second grayscale image of the preset resolution based on the preset image processing algorithm; Determining, based on the preset hash algorithm and according to pixel values of a plurality of pixels of the second grayscale image, a second hash feature of the second grayscale image; A target Hamming distance between the first hash feature and the second hash feature is calculated, and a similarity between the first feature image and the original image is determined according to the target Hamming distance.
4. The method according to claim 3, characterized in that The determining, based on a preset hash algorithm and according to pixel values of a plurality of pixels of the first grayscale image, a first hash feature of the first grayscale image includes: Traversing a plurality of pixels of the first grayscale image row by row, and for each row of pixels, comparing pixel values between any two adjacent pixels from left to right; if the pixel value of a first pixel on the left is greater than the pixel value of a second pixel on the right, determining that a comparison result between the first pixel and the second pixel is 1; If the pixel value of the first pixel on the left is less than or equal to the pixel value of the second pixel on the right, determining that the comparison result between the first pixel and the second pixel is 0; Based on a preset order, multiple comparison results of pixels in each row of the first feature image are combined to obtain the first hash feature.
5. The method according to claim 3, characterized in that The target Hamming distance between the first hash feature and the second hash feature is obtained based on the following operation: Set the initial value S of the Hamming distance to 0; Perform a first traversal to traverse multiple comparison results in the first hash feature, and for each first comparison result in the first hash feature, obtain a second comparison result at a corresponding position in the second hash feature; If the first comparison result is not equal to the second comparison result, update S with S+1; otherwise, keep S unchanged; In response to the completion of the first traversal, the updated S is output as the target Hamming distance between the first hash feature and the second hash feature.
6. The method according to any one of claims 1 to 5, characterized in that For each second feature image, fusing the second feature image with the original image to obtain a target fused image of the original image includes: In response to the original image being a single-channel image, a second pass is performed, during which the following fusion operation is performed: For each third pixel in the original image, obtaining a pixel value of the third pixel and a pixel value of a fourth pixel corresponding to the position of the third pixel in the second feature image; Calculating a first weighted average of the pixel value of the third pixel and the pixel value of the fourth pixel based on a preset first fusion weight of the original image and a preset second fusion weight of the second feature image; using the first weighted average as a target pixel value of a pixel corresponding to the third pixel in the target fused image; In response to the second traversal ending, a target fused image including a plurality of pixels having the target pixel value is output.
7. The method according to any one of claims 1 to 5, characterized in that For each second feature image, fusing the second feature image with the original image to obtain a target fused image of the original image further includes: In response to the original image being a multi-channel image, a third pass is performed, in which the following fusion operation is performed: For each fifth pixel in the original image, obtain a pixel value of any channel of the fifth pixel and a pixel value of the channel of a sixth pixel corresponding to the position of the fifth pixel in the second feature image; Calculating a second weighted average of the pixel value of the channel of the fifth pixel and the pixel value of the channel of the sixth pixel based on a preset third fusion weight of the original image and a preset fourth fusion weight of the second feature image; Using the second weighted average as the target pixel value of the channel of the pixel corresponding to the fifth pixel in the target fused image; In response to the end of the third traversal, a target fused image including a plurality of pixels having the target pixel value in each channel is output.
8. An image data enhancement device, characterized in that: include: An acquisition module, used to acquire the original image; a feature extraction module, configured to input the original image into a pre-trained convolutional neural network and obtain first feature images output by at least one convolutional layer before the first pooling layer; wherein the resolution of each first feature image is equal to the resolution of the original image; a difference comparison module, configured to determine at least one second feature image from at least one of the first feature images according to a difference between each of the first feature images and the original image; The fusion module fuses each second feature image with the original image to obtain a target fused image of the original image, wherein the target fused image is used to perform image data enhancement on the original image.
9. An image data enhancement device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.