Image processing method, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image processing method, electronic equipment and a computer readable storage medium. According to the method, the target iteration direction of each pixel point is determined according to the initial iteration direction of each pixel point in the obtained target image on different channels, and the channel energy of the target image on the sequentially arranged channels is sequentially increased or decreased; determining a target iteration step length of each pixel point according to the target iteration direction of each pixel point; and performing iteration processing on each pixel point according to the target iteration direction and the target iteration step length of each pixel point to obtain a processed target image. Therefore, the image precision is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, and computer-readable storage medium. Background Technology

[0002] Images have a wide range of applications, including biomedicine, traffic safety, and machine vision. The quality of an image directly affects its effectiveness in these applications. However, images are often affected by noise during acquisition, processing, and transmission, thus impacting image quality. Therefore, filtering noise from images is of paramount importance.

[0003] Current image processing methods typically employ filtering algorithms to reduce image noise. However, this filtering method loses a lot of useful image information during noise reduction, resulting in poor image quality after noise reduction. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide an image processing method, electronic device, and computer-readable storage medium that can improve image quality.

[0005] To address the aforementioned technical problems, this application provides an image processing method applied to an image denoising model in a deep learning network. The image denoising model includes several channels arranged in sequence. The method includes: determining the target iteration direction of each pixel in the acquired target image based on the initial iteration direction of each pixel in different channels, wherein the channel energy of the target image in the sequentially arranged channels increases or decreases sequentially; determining the target iteration step size of each pixel based on the target iteration direction of each pixel; and performing iterative processing on each pixel based on the target iteration direction and the target iteration step size to obtain the processed target image.

[0006] In one embodiment, the step of determining the target iteration direction of each pixel based on the initial iteration direction of each pixel in different channels in the acquired target image includes: determining the distance difference of each pixel in each initial iteration direction based on each adjacent pixel in each initial iteration direction; sorting each initial iteration direction of each pixel based on the distance difference of each pixel in each initial iteration direction to obtain sorted initial iteration directions. The target iteration direction for each pixel is determined based on the initial iteration direction after sorting each pixel.

[0007] In one embodiment, the target iteration direction includes the target position of the target pixel, and the step of determining the target iteration step size of each pixel based on the target iteration direction of each pixel includes: in response to the target position of the pixel being in a different channel from the target position of the target pixel in the corresponding target iteration direction, obtaining the channel energy difference between the channel where the pixel is located and the channel where the target pixel is located; and determining the target iteration step size of the pixel based on the channel energy difference.

[0008] In one embodiment, the step of iteratively processing each pixel according to the target iteration direction and the target iteration step size to obtain the processed target image includes: The current iteration pixel corresponding to each pixel is determined based on the target iteration direction and the target iteration step size; it is determined whether each current iteration pixel meets the iteration requirements based on each pixel and the corresponding current iteration pixel; if so, each current iteration pixel is determined as the pixel after each iteration; the channels of each pixel in the initial image are adjusted for each pixel after each iteration to obtain the processed target image.

[0009] In one embodiment, the step of determining whether each current iteration pixel meets the iteration requirements based on each pixel and the corresponding current iteration pixel includes: determining the iteration error based on each pixel and the corresponding current iteration pixel; and determining that each current iteration pixel meets the iteration requirements in response to the iteration error being less than a preset error threshold.

[0010] In one embodiment, before the step of determining the target iteration direction of each pixel based on the initial iteration direction of each pixel in different channels in the acquired target image, the method further includes: obtaining the pixel mean and pixel variance of each channel based on the values ​​of each pixel in different channels in the acquired initial image; obtaining the channel energy of each channel based on the pixel mean and pixel variance of each channel; and adjusting the channels of each pixel in the initial image based on the channel energy of each channel to obtain the target image.

[0011] In one embodiment, the step of adjusting the channels of each pixel in the initial image according to the channel energy of each channel to obtain the target image includes: sorting the channels according to the channel energy of each channel to obtain sorted channels; adjusting the values ​​of each pixel in the initial image on the channels before sorting to the channels after sorting to obtain the target image.

[0012] In one embodiment, the deep learning network is a second target motion detection network, which is trained as follows: a first initial motion detection network is trained based on acquired noiseless sample image pairs to obtain a first target motion detection network; noise is processed on the noiseless sample image pairs to obtain noisy sample image pairs; a second initial motion detection network is determined based on the network structure of the first target motion detection network; and the second initial motion detection network is trained based on the noisy sample image pairs and the first target motion detection network to obtain the second target motion detection network.

[0013] To address the aforementioned technical problems, this application provides an electronic device, including a memory and a processor. The memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the aforementioned image processing method.

[0014] To address the aforementioned technical problems, this application provides a computer-readable storage medium, comprising: storing program data, which, when executed by a processor, is used to implement the aforementioned image processing method.

[0015] The above scheme determines the target iteration direction of each pixel based on the initial iteration direction of each pixel in different channels of the acquired target image, with the channel energy of the target image increasing or decreasing sequentially in the ordered channels. The target iteration step size of each pixel is determined based on its target iteration direction. Iterative processing of each pixel is then performed according to its target iteration direction and target iteration step size to obtain the processed target image. Therefore, by iteratively processing each pixel using the target iteration direction and target iteration step size, the differences between pixels are reduced, thereby reducing image noise and improving image accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a schematic flowchart of an exemplary embodiment of the image processing method shown in this application; Figure 2 It is A shown in this application. 222 A schematic diagram of the data elements corresponding to each initial iteration direction on the first channel; Figure 3 It is A shown in this application. 222 A schematic diagram of the data elements corresponding to each initial iteration direction on the second channel; Figure 4 It is A shown in this application. 222 A schematic diagram of the data elements corresponding to each initial iteration direction on the third channel; Figure 5 This is a schematic diagram of the structure of an exemplary embodiment of the deep learning network shown in this application; Figure 6 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of this application; Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application; Figure 8 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] First, it's important to note that image processing has a wide range of applications, including biomedicine, traffic safety, and machine vision. Image quality directly impacts the effectiveness of these applications. However, images are frequently affected by noise during acquisition, processing, and transmission, thus affecting image quality. Therefore, filtering noise from images is of paramount importance. Current image processing methods typically employ filtering algorithms for noise reduction. However, this filtering approach, while reducing noise, inevitably loses a significant amount of useful image information, resulting in poor image quality after noise reduction.

[0019] Based on this, this application provides an image processing method, an electronic device, and a computer-readable storage medium. For details, please refer to [link / reference needed]. Figure 1 , Figure 1 This is a schematic flowchart of an exemplary embodiment of an image processing method shown in this application.

[0020] The executing entity of an image processing method can be a terminal device, a server, or other processing device. The terminal device can be a computer, mobile device, terminal, computing device, vehicle-mounted device, etc. The executing entity of an image processing method can also be an image processing device. In some possible implementations, the image processing method can be implemented by a processor calling computer-readable instructions stored in memory. The executing entity of an image processing method can also be a big data cluster. A big data cluster is a computer system architecture formed by multiple computers connected through a network. The big data cluster can be deployed on a private cloud built with K8S (Kubernetes, a container orchestration engine).

[0021] Specifically, an image processing method in this embodiment is applied to an image denoising model in a deep learning network. The image denoising model includes several channels arranged in sequence, and the image processing method includes the following steps: Step S110: Determine the target iteration direction of each pixel in the target image based on the initial iteration direction of each pixel in different channels. The channel energy of the target image in the sequentially arranged channels increases or decreases sequentially.

[0022] A target image is an image whose pixels have been rearranged from those of an initial image. In an image denoising model, the pixels in each channel of the target image cause the channel energy of each channel to increase or decrease, or the pixels in each channel of the target image cause the channel energy difference between adjacent channels to be less than a preset energy threshold. As an example, the image processing device obtains the pixel mean and pixel variance of each channel based on the pixel values ​​of each pixel in the initial image; obtains the channel energy of each channel based on the pixel mean and pixel variance; and determines the initial image as the target image if the difference in channel energy between adjacent channels is less than a preset energy threshold. As another example, the image processing device obtains the pixel mean and pixel variance of each channel based on the pixel values ​​of each pixel in the initial image; obtains the channel energy of each channel based on the pixel mean and pixel variance; sorts the channels in ascending order of channel energy; and adjusts the position of the pixel values ​​in each channel of the initial image according to the channel order to obtain the values ​​of the target image in each channel.

[0023] The initial image can be a color image captured by the imaging device, a feature map extracted by the feature extraction module, or a grayscale image. The initial image has dimensions C x H x W, where C represents the number of channels, H represents the height of the image, and W represents the width of the image.

[0024] A channel is a dimension of a feature map, representing a set of information related to a specific feature or attribute of the input image. For example, the output channel in an image denoising model.

[0025] The initial iteration direction refers to the direction between the value of each pixel in different channels and its neighboring values. Specifically, the image processing device acquires multiple neighboring values ​​of each pixel in different channels, and determines the position of each neighboring value and its corresponding neighboring value as the initial iteration direction.

[0026] The target iteration direction refers to the final iteration direction selected from the initial iteration directions. The target iteration direction includes the target position of the target pixel and the target numerical value of the target pixel at that position. For example, A 222 The target iteration direction is A 112 The target iteration direction of the value at the second row and second column position in the second channel is the value at the first row and second column position in the first channel. It can also be expressed as the target iteration direction of the pixel value at the second row and second column position in the second channel of the target image is the pixel value at the first row and second column position in the first channel of the target image.

[0027] The image processing device determines the target iteration direction for each pixel based on the initial iteration direction of each pixel in different channels of the acquired target image. Specifically, the image processing device acquires the pixel difference between the adjacent values ​​in each initial iteration direction of the pixel and the value of the corresponding pixel at the target position; sorts the initial iteration directions in ascending order of the pixel differences to obtain sorted initial iteration directions; and selects the initial iteration direction that is sorted as the target iteration direction for the corresponding pixel.

[0028] Step S120: Determine the target iteration step size of each pixel based on the target iteration direction of each pixel.

[0029] The target iteration step size refers to the magnitude of parameter updates in each iteration.

[0030] The image processing device determines the target iteration step size for each pixel based on the target iteration direction of each pixel. Specifically, in response to a pixel's target position being in the same channel as the target position of an iterated pixel in the corresponding target iteration direction, the image processing device determines a preset iteration step size as the target iteration step size. The preset iteration step size can be 1.

[0031] Step S130: Perform iterative processing on each pixel according to the target iteration direction and target iteration step size to obtain the processed target image.

[0032] The target iteration direction includes the target value of the target pixel at the target position. Iterative processing of each pixel is performed according to the target iteration direction and the target iteration step size to obtain the processed target image. Specifically, the image processing device obtains the current iteration pixel corresponding to each pixel based on the target value and the target iteration step size; it obtains the difference between the value of each current iteration pixel in each channel and the value of the previous iteration pixel in each channel. If the difference is less than a preset iteration error, it determines that each current iteration pixel meets the iteration requirements and determines each current iteration pixel as a pixel in the processed target image; if the difference is greater than or equal to the preset iteration error, it uses each current iteration pixel as a pixel in a new initial image, and performs channel adjustment on each pixel in the new initial image according to the channel energy of each channel to obtain a new target image. Then, it returns to step S110 until each iteration pixel meets the iteration requirements, and determines each iteration pixel that meets the iteration requirements as a pixel in the processed target image to obtain the processed target image.

[0033] As can be seen, the target iteration direction of each pixel in the acquired target image is determined based on the initial iteration direction of each pixel in different channels, and the channel energy of the target image increases or decreases sequentially in the ordered channels. The target iteration step size of each pixel is determined based on the target iteration direction. Iterative processing of each pixel is performed based on the target iteration direction and target iteration step size to obtain the processed target image. Therefore, by iteratively processing each pixel through the target iteration direction and target iteration step size, the difference between pixels is reduced, thereby reducing image noise and improving image accuracy.

[0034] Before determining the target iteration direction of each pixel based on the initial iteration direction of each pixel in different channels in the acquired target image, the method further includes: obtaining the pixel mean and pixel variance of each channel based on the values ​​of each pixel in different channels in the acquired initial image; obtaining the channel energy of each channel based on the pixel mean and pixel variance of each channel; and adjusting the channels of each pixel in the initial image based on the channel energy of each channel to obtain the target image.

[0035] The average pixel value of a pixel in a channel satisfies the following formula:

[0036] In the above formula, Characterizing the first t The pixel mean of each channel, where t ranges from 1 to C, D Characterizing the first t The value in the i-th row and j-th column of each channel.

[0037] The pixel variance of a pixel in a channel satisfies the following formula:

[0038] In the above formula, Characterizing the first t The pixel variance of each channel.

[0039] The image processing device obtains the channel energy of each channel based on the pixel mean and pixel variance of each channel. Specifically, the image processing device determines the channel energy of a channel as the first product term between the pixel mean and pixel variance of the channel.

[0040] For example, the channel energy of a channel satisfies the following formula:

[0041] In the above formula, Characterizing the first t Channel energy of each channel.

[0042] The image processing device performs channel adjustment on each pixel in the initial image based on the channel energy of each channel to obtain the target image. As an example, the image processing device adjusts the position of each channel in ascending order of channel energy to obtain the adjusted channels, and then adjusts the position of each value according to the channel adjustment method to obtain the value of each pixel in each channel in the target image.

[0043] As another example, the step of adjusting the channels of each pixel in the initial image according to the channel energy of each channel to obtain the target image includes: sorting the channels according to the channel energy of each channel to obtain sorted channels; adjusting the values ​​of each pixel in the initial image on the channels before sorting to the channels after sorting to obtain the target image.

[0044] The data processing device sorts the channels according to their energy levels, resulting in a sorted list of channels. Specifically, the data processing device sorts the channels in ascending order of energy level, resulting in a sorted list of channels.

[0045] The data processing device adjusts the values ​​of each pixel in the initial image from the channels before sorting to the channels after sorting, thus obtaining the target image. Specifically, the data processing device adjusts the values ​​of each channel before sorting to their sorted positions along with the channels. For example, if a channel was third before sorting and is now second after sorting, the original third channel is moved to the second channel, and the values ​​of the original third channel are moved entirely to the second channel.

[0046] The step of determining the target iteration direction of each pixel in the acquired target image based on the initial iteration direction of each pixel in different channels includes: determining the distance difference of each pixel in each initial iteration direction based on each adjacent pixel in each initial iteration direction; sorting the initial iteration directions of each pixel based on the distance difference of each pixel in each initial iteration direction to obtain sorted initial iteration directions; and determining the target iteration direction of each pixel based on the sorted initial iteration directions of each pixel.

[0047] The image processing device determines the distance difference between each pixel in each initial iteration direction based on the distance between each pixel and its neighboring pixels in each initial iteration direction. Specifically, the image processing device obtains a first sum value between neighboring pixels in the first initial iteration direction and a second sum value between neighboring pixels in the second initial iteration direction; the difference between the first sum value and the second sum value is determined as the distance difference, wherein the first initial iteration direction is opposite to the second initial iteration direction.

[0048] In this context, the numerical values ​​of each pixel in each channel of the target image are represented by data element A. abc In this representation, 'a' represents the number of channels, 'b' represents the height, and 'c' represents the width. For example, data element A... 222 Represents the value at a height of 2 and a width of 2 in the second channel, or the value in the second row and second column of the second channel, and A. 222 The data elements corresponding to each initial iteration direction on the first channel adjacent to the current channel are as follows: Figure 2 As shown, A 222 Data elements corresponding to each initial iteration direction on the second channel are as follows Figure 3 As shown, with A 222 The data elements corresponding to each initial iteration direction on the third channel adjacent to the current channel are as follows: Figure 4 As shown, each data element in its own channel and in each of the two adjacent channels in each initial iteration direction is grouped into a 3x3x3 neighborhood. If a data element is located at an edge, the 3x3x3 neighborhood is formed by mirroring and / or padding with zeros.

[0049] Each initial iteration direction may include data elements, such as data element A. 222 The initial iteration direction includes A 221 Then the value in the second row and first column of the second channel represents data element A. 222 The initial iteration direction.

[0050] In one embodiment, such as Figure 2 , Figure 3 and Figure 4 As shown, data element A 222The steps for calculating the distance difference for each of the 26 initial iteration directions, which are opposite to each other, include: The initial iteration direction is A 221 With the initial iteration direction being A 223 Distance difference between This can be expressed as the following formula:

[0051] The initial iteration direction is A 212 With the initial iteration direction being A 232 Distance difference between This can be expressed as the following formula:

[0052] The initial iteration direction is A 211 With the initial iteration direction being A 233 Distance difference between This can be expressed as the following formula:

[0053] The initial iteration direction is A 213 With the initial iteration direction being A 231 Distance difference between This can be expressed as the following formula:

[0054] The initial iteration direction is A 111 With the initial iteration direction being A 333 Distance difference between This can be expressed as the following formula:

[0055] The initial iteration direction is A 113 With the initial iteration direction being A 331 Distance difference between This can be expressed as the following formula:

[0056] The initial iteration direction is A 133 With the initial iteration direction being A 311 Distance difference between This can be expressed as the following formula:

[0057] The initial iteration direction is A 131 With the initial iteration direction being A 313 Distance difference between This can be expressed as the following formula:

[0058] The initial iteration direction is A 112 With the initial iteration direction being A 332 Distance difference between This can be expressed as the following formula:

[0059] The initial iteration direction is A 123 With the initial iteration direction being A 321 Distance difference between This can be expressed as the following formula:

[0060] The initial iteration direction is A 132 With the initial iteration direction being A 312 Distance difference between This can be expressed as the following formula:

[0061] The initial iteration direction is A 121 With the initial iteration direction being A 323 Distance difference between This can be expressed as the following formula:

[0062] The initial iteration direction is A 122 With the initial iteration direction being A 322 Distance difference between This can be expressed as the following formula:

[0063] The image processing device sorts the initial iteration directions of each pixel according to the distance difference between each pixel in each initial iteration direction, resulting in sorted initial iteration directions. Specifically, the image processing device sorts the initial iteration directions in ascending order of the absolute value of each distance difference, resulting in sorted initial iteration directions.

[0064] The image processing device determines the target iteration direction for each pixel based on the initial iteration directions after the pixels are sorted. Specifically, the image processing device selects the first two initial iteration directions from the initial iteration directions after the pixels are sorted to obtain the candidate iteration directions for the corresponding pixel; obtains the distance difference between the pixel on each candidate iteration direction and the corresponding pixel; and selects the candidate iteration direction with the smallest distance difference as the target iteration direction for the corresponding pixel.

[0065] In one embodiment, for | |to| Compare the results to find the minimum value. Then A 112 With A 332 The initial iteration direction is used as a candidate iteration direction; then compare... and The size, if Less than Then A 112 As for A 222 The target iteration direction.

[0066] The target iteration direction includes the target position of the target pixel. The step of the image processing device determining the target iteration step size of each pixel according to the target iteration direction of each pixel includes: in response to the target position of the pixel being in a different channel from the target position of the target pixel in the corresponding target iteration direction, obtaining the channel energy difference between the channel where the pixel is located and the channel where the target pixel is located; and determining the target iteration step size of the pixel according to the channel energy difference.

[0067] The target location of a target pixel includes the channel in which the target pixel is located, as well as its height and width within that channel.

[0068] The image processing device acquires the channel energy difference between the channel where the pixel is located and the channel where the target pixel is located. Specifically, the image processing device determines the channel energy difference as the difference between the channel energy of the channel where the pixel is located and the channel energy of the corresponding target pixel.

[0069] For example, if the channel of a pixel is the t-th channel and the channel of the target pixel is the (t+1)-th channel, then the channel energy difference satisfies the following formula:

[0070] In the above formula, dE Characterizing the channel energy difference, E t+1 The channel energy of the (t+1)th channel is represented. E t This represents the channel energy of the t-th channel. It should be noted that the larger the channel energy difference, the greater the energy gap between the two channels, and therefore the lower the similarity between the two channels.

[0071] The image processing device determines the target iteration step size of a pixel based on the channel energy difference. Specifically, the image processing device determines the channel iteration intensity based on the comparison between the absolute value of the channel energy difference and a preset maximum energy threshold; in response to the pixel's position on the channel being consistent with the target pixel's position on the channel, the channel iteration intensity is determined as the target iteration step size; in response to the pixel's position on the channel being inconsistent with the target pixel's position on the channel, the target iteration step size is calculated based on the channel iteration intensity.

[0072] The image processing device determines the channel iteration intensity based on a comparison between the absolute value of the channel energy difference and a preset maximum energy threshold. Specifically, in response to the absolute value of the channel energy difference being less than the preset maximum energy threshold, the image processing device obtains the difference between the preset maximum energy threshold and the channel energy difference, and determines the ratio between this difference and the preset maximum energy threshold as the channel iteration intensity; in response to the absolute value of the channel energy difference being greater than or equal to the preset maximum energy threshold, the preset iteration intensity is determined as the channel iteration intensity.

[0073] The channel iteration strength can be expressed by the following formula:

[0074] In the above formula, sE Characterizes the channel iteration intensity. maxdE This represents a preset maximum energy threshold. When the absolute value of the channel energy difference is greater than or equal to the preset maximum energy threshold, the feature difference between channels is too large, so no iteration is performed between channels, and the channel iteration intensity is set to 0.

[0075] The image processing device determines the channel iteration intensity as the target iteration step size in response to the fact that the position of a pixel on the channel is consistent with the position of the target pixel on the channel; in response to the fact that the position of a pixel on the channel is inconsistent with the position of the target pixel on the channel, it calculates the target iteration step size based on the channel iteration intensity.

[0076] Specifically, the image processing device determines whether the height and width of the pixel are the same as the height and width of the target pixel. If so, it determines that the position of the pixel on the channel is the same as the position of the target pixel on the channel; otherwise, it determines that the position of the pixel on the channel is not the same as the position of the target pixel on the channel. Alternatively, the image processing device determines whether the row and column of the pixel are the same as the row and column of the target pixel. If so, it determines that the position of the pixel on the channel is the same as the position of the target pixel on the channel; otherwise, it determines that the position of the pixel on the channel is not the same as the position of the target pixel on the channel.

[0077] For example, when the channel of a pixel is different from the channel of the target pixel, but the position of the pixel on the channel is the same as the position of the target pixel on the channel, the pixel and the target pixel only have inter-channel offset, and the channel iteration intensity and the target iteration step size satisfy the following formula:

[0078] In the above formula, step Characterize the target iteration step size.

[0079] In one embodiment, data element A 222 The data element in the target iteration direction is A 122 It can be seen that A 222 Located in the second row and second column of the second channel, A 122 If A is located in the second row and second column of the first channel, then A 222 With A 122 For channels that are in different channels but have the same position on the channel, the channel iteration intensity is determined as the target iteration step size.

[0080] For example, when the channel of a pixel is the same as the channel of the target pixel, but the position of the pixel in the channel is different from the position of the target pixel in the channel, the pixel and the target pixel only have an in-channel offset, and the target iteration step size satisfies the following formula:

[0081] In the above formula, The preset iteration step size can be 1.

[0082] In one embodiment, data element A 222 The data element in the target iteration direction is A 212 It can be seen that A 222 Located in the second row and second column of the second channel, A 212 If A is located in the first row and second column of the second channel, then A 222 With A 212 For items on the same channel but at different positions within the channel, the preset iteration step size is determined as the target iteration step size.

[0083] For example, when the channel of a pixel is different from the channel of the target pixel, and the position of the pixel on the channel is also different from the position of the target pixel on the channel, there is inter-channel and intra-channel offset between the pixel and the target pixel. The channel iteration intensity and the target iteration step size satisfy the following formula:

[0084] In the above formula, step Characterize the target iteration step size.

[0085] In one embodiment, data element A 222 The data element in the target iteration direction is A 112 It can be seen that A 222 Located in the second row and second column of the second channel, A 112 If A is located in the first row and second column of the first channel, then A 222 With A 112 The target iteration step size is calculated based on the iteration intensity of the channels, since the channels are in different channels and their positions are also different.

[0086] It can be seen that the target position of a pixel is in a different channel than the target position of the target pixel in the corresponding target iteration direction. Therefore, the channel energy difference between the channel where the pixel is located and the channel where the target pixel is located is obtained. The target iteration step size of the pixel is determined based on the channel energy difference. Thus, determining the target iteration step size by using the channel energy difference can reduce the iteration step size between channels when the energy difference between channels is large, thereby improving the iteration effect.

[0087] The image processing device performs iterative processing on each pixel according to the target iteration direction and target iteration step size to obtain the processed target image. The steps include: determining the current iteration pixel corresponding to each pixel based on the target iteration direction and target iteration step size; determining whether each current iteration pixel meets the iteration requirements based on each pixel and the corresponding current iteration pixel; if so, determining each current iteration pixel as the pixel after iteration; and adjusting the channels of each pixel after iteration based on the channels of each pixel in the initial image to obtain the processed target image.

[0088] The image processing device determines the current iteration pixel corresponding to each pixel based on the target iteration direction and the target iteration step size. Specifically, the image processing device obtains the third sum between the value of the target pixel in the channel and the value of the pixel in the channel; obtains the second product term between the third sum and the target iteration step size; obtains the third product term between the second product term and a preset adjustment parameter; and determines the fourth sum between the value of the pixel in the channel and the third product term as the value of the current iteration pixel in the channel.

[0089] For example, the value of the pixel in the second row and second column on the second channel is represented by data element A. 222 A 222 The data element corresponding to the target element in the target iteration direction is A. 112 Then the data element corresponding to the current iteration pixel satisfies the following formula:

[0090] In the above formula, The data element representing the pixel in the current iteration. Characterizes the preset adjustment parameters. These preset adjustment parameters are used to adjust the iteration step size, and their values ​​range from 0 to 1. For example, =0.2.

[0091] The step of the image processing device determining whether each current iteration pixel meets the iteration requirements based on each pixel and the corresponding current iteration pixel includes: determining the iteration error based on each pixel and the corresponding current iteration pixel; and determining that each current iteration pixel meets the iteration requirements in response to the iteration error being less than a preset error threshold.

[0092] The image processing device determines the iteration error based on each pixel and the corresponding current iteration pixel. Specifically, the image processing device determines the iteration error as the Euclidean norm between the value of each pixel in each channel and the value of the current iteration pixel in each channel.

[0093] It should be noted that each pixel can be the pixel of the m-th iteration, and the corresponding pixel of the current iteration can be the pixel of the (m+1)-th iteration. Therefore, the pixel in the (m+1)-th iteration satisfying the iteration requirement can be expressed by the following formula:

[0094] In the above formula, The value of the pixel in each channel represents the value of the pixel in the m-th iteration. This represents the value of the pixel in each channel during the (m+1)th iteration. Characterizes the preset error threshold. It can be 0.1.

[0095] The image processing device adjusts the channels of each pixel in the initial image for each iteration to obtain the processed target image. Specifically, the image processing device adjusts each channel in the target image back to its original position according to the channel order in the initial image, and adjusts each iterated pixel to its original channel position according to its channel position in the initial image to obtain the processed target image.

[0096] In one embodiment, for example, after channel adjustment of each pixel in the initial image, the second channel becomes the first channel. Then, the values ​​of each pixel in the initial image in the second channel are adjusted to the values ​​of each pixel in the target image in the first channel. After iterative processing of the values ​​of each pixel in the first channel, the iterative values ​​of each pixel in the first channel are obtained. After converting the first channel back to the second channel, the iterative values ​​of each pixel in the first channel are adjusted to the second channel.

[0097] As can be seen, the image processing device adjusts each channel in the target image back to its original position according to the channel order in the initial image, and adjusts each iterated pixel to its original channel position according to its channel position in the initial image, thus obtaining the processed target image. This restores the channels to their original order, ensuring the normal operation of the model.

[0098] The image processing device further includes: determining whether each current iteration pixel meets the iteration requirements based on each pixel and the corresponding current iteration pixel; if not, taking each current iteration pixel as a pixel in the new initial image, adjusting the channels of each pixel in the new initial image to obtain a new target image, and returning to step S110.

[0099] In one embodiment, the image denoising model can be applied to any module of a deep learning network, such as ResNet (Residual Network), UNet (U-shaped network), and VGG16 (Visual Geometry Group 16-layer network). Figure 5 As shown, a deep learning network can include an image denoising model and a feature extraction module. The feature extraction module includes convolutional layers, normalization layers, and activation layers.

[0100] It's important to note that video denoising is a crucial aspect of video image processing. Motion detection within video is a key module in video denoising. With the development of deep learning, most video image processing chips now incorporate deep learning processors, and deep learning outperforms traditional algorithms in processing video images. Consequently, deep learning is increasingly replacing traditional video image processing algorithms.

[0101] Deep learning networks can be used for motion detection. Training motion detection networks requires a large number of sample pairs, and their final performance in practical applications is strongly correlated with the noise distribution of the training samples. If the noise distribution of the training samples differs significantly from the noise distribution in actual use, the actual performance of the motion detection network will be poor. Different image sensors acquire images with different noise distributions, and subsequent image processing algorithms further alter this noise distribution, making training motion detection networks quite difficult. This application addresses this issue by having the motion detection network learn the signal rather than the noise distribution during training, reducing noise perturbation to the feature images, lowering the difficulty of model training, and improving image accuracy. Furthermore, it incorporates an image denoising model into the motion detection network to iteratively denoise the images, further improving image accuracy.

[0102] The deep learning network is a second target motion detection network, which is trained as follows: a first initial motion detection network is trained based on the acquired noiseless sample image pairs to obtain a first target motion detection network; noise is processed on the noiseless sample image pairs to obtain noisy sample image pairs; a second initial motion detection network is determined based on the network structure of the first target motion detection network; and the second initial motion detection network is trained based on the noisy sample image pairs and the first target motion detection network to obtain the second target motion detection network.

[0103] The first and second target motion detection networks include image denoising models. Specifically, after the image processing device inputs the initial image into the first or second target motion detection network, it performs denoising processing on the initial image using the image denoising model to obtain the processed target image.

[0104] A motion detection network can be represented by the following formula:

[0105] In the above formula, inFrame 1 and inFrame 2 represents the first and second adjacent frames in a video. Characterize network parameters, F Characterizing motion detection networks, output This represents the motion detection result, i.e., the output of the motion detection network. The intermediate layers of the motion detection network include f1, f2, ..., fn.

[0106] The image processing device trains a first initial motion detection network based on the acquired noiseless sample image pairs to obtain a first target motion detection network. Specifically, the image processing device inputs each noiseless sample image pair into the first initial motion detection network to obtain a first output result; calculates a first loss value between the first output result and the actual motion state of the noiseless sample image pair; and trains the first initial motion detection network with the goal of convergence of the first loss value until the first loss value meets a preset loss requirement to obtain the first target motion detection network.

[0107] A noise-free sample image pair refers to two frames of images without noise. Specifically, the image processing device extracts two adjacent frames from the video, performs noise-free processing on the two adjacent frames to obtain noise-free adjacent images; it then performs motion state annotation on each region in the noise-free adjacent images to obtain adjacent images with label masks; and finally, the adjacent images with label masks are identified as noise-free sample image pairs. Motion state annotation refers to labeling moving regions in the image as 1 and stationary regions as 0.

[0108] The image processing device further includes: performing augmentation processing on each noise-free sample image pair using a preset data augmentation method to obtain multiple noise-free sample image pairs. The augmentation processing can include geometric transformation, color transformation, blurring, and sharpening, etc.

[0109] The first target motion detection network can be represented by the following formula:

[0110] In the above formula, The network parameters characterize the motion detection network for the first target. F Characterizing motion detection networks, output This is the output.

[0111] The image processing device performs noise processing on noiseless sample image pairs to obtain noisy sample image pairs. Specifically, preset noise is added to the noiseless sample images to obtain noisy sample image pairs.

[0112] In one embodiment, the image processing device adds preset noise to each noiseless sample image pair to obtain noisy sample image pairs. The preset noise can be of different sizes and types.

[0113] The image processing device determines a second initial motion detection network based on the network structure of the first target motion detection network. Specifically, the image processing device uses the network structure of the first target motion detection network as the network structure of the second initial motion detection network.

[0114] The second initial motion detection network can be expressed by the following formula:

[0115] In the above formula, Characterizes the network structure of the second initial motion detection network. and F Exactly the same.

[0116] The image processing device trains a second initial motion detection network based on noisy sample images to obtain a second target motion detection network. Specifically, the first target motion detection network includes multiple first processing layers, and the second initial motion detection network includes multiple second processing layers. The image processing device inputs noisy sample image pairs into the second initial motion detection network to obtain the processing results of each second processing layer; it inputs noiseless sample image pairs into the first target motion detection network to obtain the processing results of each first processing layer; it calculates a second loss value based on the processing results of each second processing layer and the processing results of each first processing layer; and it trains the second initial motion detection network with the goal of convergence of the second loss value to obtain the second target motion detection network.

[0117] The second loss value satisfies the following formula:

[0118] In the above formula, loss Characterizing the second loss value, The representation of the 1-norm, f 1, f 2, ..., f n For the first target motion detection network The processing results of each of the first processing layers in the middle. t 1, t 2, ..., t n For the second initial motion detection network The processing results of the second processing layer in the middle, where The first noisy image in a pair of noisy sample images is represented. The second noisy image in the noisy sample image pair. This represents the sum of the absolute values ​​of the subtraction between corresponding elements in the processing results of the first and second processing layers. For example, if the processing layers are in the form of tensors (i.e., three-dimensional data), then... This represents the sum of the absolute values ​​of the subtraction of corresponding elements of two tensors.

[0119] In this application, when performing motion detection on real noisy videos using the first target motion detection network, the noise distribution in the training samples is inconsistent with that in the real videos, as the original training samples are noise-free. This inconsistent noise distribution can perturb the intermediate layers of the network, affecting the accuracy of the final motion detection. Therefore, this application uses the first target motion detection network to learn motion judgment from noise-free training samples and the second target motion detection network to learn motion judgment from noisy training samples. Simultaneously, the processing layers and output of the second target motion detection network are made to approximate those of the first target motion detection network. This allows the second target motion detection network to learn the signal from the first target motion detection network, rather than the noise distribution, thereby enabling accurate motion detection even in noisy images.

[0120] In one embodiment, the second target motion detection network provided in this application differs from the teacher-student network in the following three ways. First, the teacher-student networks have different network structures, and the teacher network is typically much larger than the student network. In contrast, the first and second target motion detection networks in this application have the same network structure, reducing the difficulty of model training. Second, both teacher and student networks use noisy samples, while the first target motion detection network in this application uses noiseless images for training. Third, the teacher network needs to be trained to a high level of quality, therefore the noise distribution of the training samples should be as close as possible to the noise distribution of the real data. However, the second target motion detection network in this application learns the signal from the first target motion detection network, so it only needs that the noise in the sample images has diversity, and it is not required to be consistent with the noise distribution of the real data. That is, there are no requirements for the noise distribution in the sample images, reducing the requirements for training data.

[0121] Figure 6 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of this application. Figure 6 As shown, the exemplary image processing apparatus 600 includes: a target iteration direction determination module 610, a target iteration step size determination module 620, and an iteration processing module 630. Specifically: The target iteration direction determination module 610 is used to determine the target iteration direction of each pixel in the acquired target image based on the initial iteration direction of each pixel in different channels. The channel energy of the target image in the sequentially arranged channels increases or decreases sequentially.

[0122] The target iteration step size determination module 620 is used to determine the target iteration step size of each pixel based on the target iteration direction of each pixel.

[0123] The iterative processing module 630 is used to perform iterative processing on each pixel according to the target iteration direction and the target iteration step size to obtain the processed target image.

[0124] In this exemplary image processing apparatus, the target iteration direction of each pixel is determined based on the initial iteration direction of each pixel in different channels of the acquired target image, and the channel energy of the target image increases or decreases sequentially in the sequentially arranged channels. The target iteration step size of each pixel is determined based on its target iteration direction. Iterative processing of each pixel is performed according to its target iteration direction and target iteration step size to obtain the processed target image. Thus, by iteratively processing each pixel using the target iteration direction and target iteration step size, the differences between pixels are reduced, thereby reducing image noise and improving image accuracy.

[0125] The functions of each module can be found in the image processing method implementation examples, and will not be repeated here.

[0126] To implement the image processing method of the above embodiments, this application proposes another electronic device, please refer to [link / reference needed]. Figure 7 , Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application.

[0127] Electronic device 700 includes memory 701 and processor 702, wherein memory 701 and processor 702 are coupled together.

[0128] The memory 701 is used to store program data, and the processor 702 is used to execute the program data to implement the image processing method of the above embodiment.

[0129] In this embodiment, processor 702 can also be referred to as CPU (Central Processing Unit). Processor 702 may be an integrated circuit chip with signal processing capabilities. Processor 702 can also be a general-purpose processor, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 702 can be any conventional processor.

[0130] This application also provides a computer-readable storage medium, such as Figure 8 As shown, the computer-readable storage medium 800 is used to store program data 801, which, when executed by a processor, is used to implement the image processing method as described in the method embodiments of this application.

[0131] The methods involved in the image processing method embodiments of this application, when implemented as software functional units and sold or used as independent products, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image processing method, characterized in that, An image denoising model applied to deep learning networks, the image denoising model comprising a plurality of channels arranged in sequence, the method comprising: The target iteration direction of each pixel is determined based on the initial iteration direction of each pixel in different channels in the acquired target image, and the channel energy of the target image in the sequentially arranged channels increases or decreases sequentially. The target iteration step size of each pixel is determined based on the target iteration direction of each pixel. The target image is obtained by iterating each pixel according to the target iteration direction and the target iteration step size.

2. The method according to claim 1, characterized in that, The step of determining the target iteration direction of each pixel based on the initial iteration direction of each pixel in different channels of the acquired target image includes: The distance difference between each pixel in each initial iteration direction is determined based on the distance between each pixel and its neighboring pixels in each initial iteration direction. The initial iteration directions of each pixel are sorted according to the distance difference of each pixel in each initial iteration direction, and the sorted initial iteration directions are obtained. The target iteration direction for each pixel is determined based on the initial iteration direction after sorting each pixel.

3. The method according to claim 1, characterized in that, The target iteration direction includes the target position of the target pixel, and the step of determining the target iteration step size of each pixel based on the target iteration direction of each pixel includes: In response to the fact that the target position of the pixel and the target position of the target pixel in the corresponding target iteration direction are in different channels, the channel energy difference between the channel where the pixel is located and the channel where the target pixel is located is obtained; The target iteration step size of the pixel is determined based on the channel energy difference.

4. The method according to claim 1, characterized in that, The step of iteratively processing each pixel according to the target iteration direction and the target iteration step size to obtain the processed target image includes: The current iteration pixel corresponding to each pixel is determined based on the target iteration direction of each pixel and the target iteration step size. Determine whether each current iteration pixel meets the iteration requirements based on each pixel and its corresponding current iteration pixel. If so, then each current iteration pixel will be determined as the pixel of each iteration; Based on the channels of each pixel in the initial image, the channels of each iterated pixel are adjusted to obtain the processed target image.

5. The method according to claim 4, characterized in that, The step of determining whether each current iteration pixel meets the iteration requirements based on each pixel and its corresponding current iteration pixel includes: The iteration error is determined based on each pixel and the corresponding current iteration pixel. In response to the iteration error being less than a preset error threshold, it is determined that each current iteration pixel meets the iteration requirements.

6. The method according to claim 1, characterized in that, Before the step of determining the target iteration direction of each pixel based on the initial iteration direction of each pixel in different channels of the acquired target image, the method further includes: The mean pixel value and variance of each channel are obtained based on the values ​​of each pixel in the initial image in different channels. The channel energy of each channel is obtained based on the mean pixel value and the variance of each channel. The target image is obtained by adjusting the channels of each pixel in the initial image according to the channel energy of each channel.

7. The method according to claim 6, characterized in that, The step of adjusting the channels of each pixel in the initial image according to the channel energy of each channel to obtain the target image includes: The channels are sorted according to their energy levels to obtain the sorted channels. The values ​​of each pixel in the initial image are adjusted from the channels before sorting to the channels after sorting to obtain the target image.

8. The method according to claim 1, characterized in that, The deep learning network is a second target motion detection network, which is trained in the following way: The first initial motion detection network is trained based on the acquired noise-free sample images to obtain the first target motion detection network; The noise-free sample image pairs are subjected to noise processing to obtain noisy sample image pairs; The second initial motion detection network is determined based on the network structure of the first target motion detection network; The second initial running detection network is trained based on the noisy sample image pairs and the first target motion detection network to obtain the second target motion detection network.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to perform the method as claimed in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, include: The system stores program data, which, when executed by a processor, is used to implement the method as described in any one of claims 1-8.

Images