Image processing method, device, storage medium and electronic device
Through the combination of same-channel convolution processing, cross-channel convolution processing and residual network, the bad point pixels in the image are accurately detected and corrected, solving the problem of low recovery accuracy in the prior art and significantly improving the image quality.
Patent Information
- Application Number
- CN202011134942.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-21
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-10-21
AI Technical Summary
In the prior art, electronic devices have poor accuracy in restoring pixels of image bad points, resulting in poor image quality.
Same-channel convolution processing and cross-channel convolution processing are combined with residual networks. Through feature stitching and multi-layer network processing, whether the pixel is a bad point pixel and correct it.
Improve the accurate detection and recovery ability of pixels of bad points in image and improve image quality.
Smart Images

Figure CN114387171B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image technology, and in particular relates to an image processing method, device, storage medium and electronic device. Background Art
[0002] Bad pixels refer to the process defects of the array formed by the light collection points (pixels) on the image sensor, or errors in the process of converting light signals into electrical signals, which will cause errors in pixel information on the image and inaccurate pixel values in the image. These defective pixels are bad pixels. In the related art, bad pixels on the image can be restored. However, in the related art, the accuracy of electronic equipment in restoring bad pixels is poor. Summary of the invention
[0003] The embodiments of the present application provide an image processing method, device, storage medium and electronic device, which can improve the accuracy of restoring bad pixels of an image.
[0004] In a first aspect, an embodiment of the present application provides an image processing method, comprising:
[0005] acquiring a first image;
[0006] Performing same-channel convolution processing on the first image to obtain a second image, and performing cross-channel convolution processing on the first image to obtain a third image;
[0007] Performing feature stitching on the second image and the third image to obtain a fourth image;
[0008] Processing the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel;
[0009] When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel;
[0010] Correct the bad pixel.
[0011] In a second aspect, an embodiment of the present application provides an image processing device, including:
[0012] An acquisition module, used for acquiring a first image;
[0013] A processing module, used to perform same-channel convolution processing on the first image to obtain a second image, and to perform cross-channel convolution processing on the first image to obtain a third image; to perform feature stitching on the second image and the third image to obtain a fourth image; to process the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel; when the probability value is greater than a preset threshold, to detect the pixel as a bad pixel; and to correct the bad pixel.
[0014] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the process in the image processing method provided by the embodiment of the present application.
[0015] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the processor is used to execute the process in the image processing method provided in the embodiment of the present application by calling a computer program stored in the memory.
[0016] In an embodiment of the present application, the electronic device may perform same-channel convolution processing on the first image to obtain a second image, and perform cross-channel convolution processing on the first image to obtain a third image. Afterwards, the electronic device may perform feature stitching on the second image and the third image to obtain a fourth image, and use at least one residual network to process the fourth image to determine the probability value of whether the pixel in the first image is a bad pixel. When the probability value is greater than a preset threshold, the electronic device may detect the corresponding pixel as a bad pixel. Afterwards, the electronic device may correct the detected bad pixel. Since the embodiment of the present application performs same-channel convolution processing and cross-channel convolution processing on the first image, and uses the residual network to process the image to determine the probability value of whether the pixel in the first image is a bad pixel. Therefore, the embodiment of the present application can accurately detect bad pixels and correct them, thereby improving the accuracy of the electronic device in restoring bad pixels. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.
[0018] Figure 1 This is a first flow chart of the image processing method provided in the embodiment of the present application.
[0019] Figure 2 This is a schematic diagram of the first structure of the pre-trained image restoration model provided in an embodiment of the present application.
[0020] Figure 3 It is a structural diagram of the same-channel convolution module provided in an embodiment of the present application.
[0021] Figure 4 It is a schematic diagram of the same-channel convolution kernel and the cross-channel convolution kernel provided in the embodiments of the present application.
[0022] Figure 5 It is a structural diagram of the cross-channel convolution module provided in an embodiment of the present application.
[0023] Figure 6 It is a structural diagram of the residual network module provided in an embodiment of the present application.
[0024] Figure 7 This is a second flow chart of the image processing method provided in the embodiment of the present application.
[0025] Figure 8 This is a second structural diagram of the pre-trained image restoration model provided in an embodiment of the present application.
[0026] Fig. 9 This is a third flow chart of the image processing method provided in the embodiment of the present application.
[0027] Fig.10 This is a third structural diagram of the pre-trained image restoration model provided in an embodiment of the present application.
[0028] Fig.11 It is a structural diagram of the deep learning bad pixel recovery module provided in an embodiment of the present application.
[0029] Fig.12 This is a schematic diagram of the first structure of the training model provided in the embodiment of the present application.
[0030] Fig.13 This is a second structural diagram of the training model provided in the embodiment of the present application.
[0031] Fig.14 It is a structural schematic diagram of the image processing device provided in an embodiment of the present application.
[0032] Fig.15 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0033] Fig.16 It is another structural schematic diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0034] Please refer to the drawings, where the same component symbols represent the same components. The principle of the present application is illustrated by implementing it in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, which should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.
[0035] It is understandable that the execution subject of the embodiments of the present application may be an electronic device such as a smart phone or a tablet computer.
[0036] See also Figure 1 , Figure 1 : is a schematic diagram of the process of the image processing method provided in the embodiment of the present application, and the process may include:
[0037] 101. Acquire a first image.
[0038] Bad pixels refer to the process defects of the array formed by the light collection points (pixels) on the image sensor, or errors in the process of converting light signals into electrical signals, which will cause errors in pixel information on the image and inaccurate pixel values in the image. These defective pixels are bad pixels. In the related art, bad pixels on the image can be restored. However, in the related art, the accuracy of electronic equipment in restoring bad pixels is poor.
[0039] In the embodiment of the present application, for example, the electronic device may first obtain an image, which may be recorded as a first image. It is understandable that the first image is the image that needs to be detected and corrected for bad pixels.
[0040] 102. Perform same-channel convolution processing on the first image to obtain a second image, and perform cross-channel convolution processing on the first image to obtain a third image.
[0041] 103. Perform feature stitching on the second image and the third image to obtain a fourth image.
[0042] 104. Process the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel.
[0043] 105. When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel.
[0044] For example, 102, 103, 104, and 105 include:
[0045] After acquiring the first image, the electronic device may perform same-channel convolution processing on the first image to obtain a second image, and the electronic device may also perform cross-channel convolution processing on the first image to obtain a third image.
[0046] Afterwards, the electronic device may perform feature stitching on the second image and the third image to obtain a fourth image.
[0047] After obtaining the fourth image, the electronic device may process the fourth image using at least one residual network to determine the probability value of whether the pixel in the first image is a bad pixel. For example, when the fourth image is processed using the residual network, the probability value of whether the pixel is a bad pixel may be output for each pixel in the first image. When the probability value is greater than a preset threshold, the electronic device may detect the corresponding pixel as a bad pixel.
[0048] 106. Correct bad pixels.
[0049] For example, after detecting bad pixels from the first image, the electronic device may correct the bad pixels.
[0050] It can be understood that in an embodiment of the present application, the electronic device can perform same-channel convolution processing on the first image to obtain a second image, and perform cross-channel convolution processing on the first image to obtain a third image. Afterwards, the electronic device can perform feature stitching on the second image and the third image to obtain a fourth image, and use at least one residual network to process the fourth image to determine the probability value of whether the pixel in the first image is a bad pixel. When the probability value is greater than a preset threshold, the electronic device can detect the corresponding pixel as a bad pixel. Afterwards, the electronic device can correct the detected bad pixel. Since the embodiment of the present application performs same-channel convolution processing and cross-channel convolution processing on the first image, and uses the residual network to process the image to determine the probability value of whether the pixel in the first image is a bad pixel. Therefore, the embodiment of the present application can accurately detect bad pixels and correct them, thereby improving the accuracy of the electronic device in restoring bad pixels.
[0051] In one implementation, the process from 102 to 106 may be completed by a pre-trained image restoration model, and the pre-trained image restoration model may be a network model that has been trained.
[0052] The pre-trained image restoration model can be used to detect and correct bad pixel pixels in an image. The pre-trained image restoration model may include at least a same-channel convolution module, a cross-channel convolution model, a residual network module, and a bad pixel correction module. The same-channel convolution module and the cross-channel convolution module may be connected in parallel and then connected to the residual network module. The residual network module may be connected to the bad pixel correction module, such as Figure 2 shown.
[0053] In an embodiment of the present application, the same-channel convolution module, cross-channel convolution module and residual network module in the trained model can be used to detect bad pixel. Among them, the same-channel convolution module is used to perform same-channel convolution processing on the first image to obtain the second image, the cross-channel convolution module is used to perform cross-channel convolution processing on the first image to obtain the third image, and the residual network module is used to process the fourth image using at least one residual network to determine the probability value of whether the pixel in the first image is a bad pixel. When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel. The bad pixel correction module can be used to correct the detected bad pixel.
[0054] Therefore, after acquiring the first image and the pre-trained image restoration model, the electronic device can input the first image into the pre-trained image restoration model, and the pre-trained image restoration model can detect and correct bad pixels in the first image, and output the corrected image.
[0055] In one embodiment, Figure 3 As shown, the same-channel convolution module may include a first input feature module, a first network module, a second network module, a third network module, a second network module and a first output feature module connected in sequence.
[0056] In the same channel convolution module, the first network module may include a 3×3 convolution layer with a convolution expansion coefficient of 2 (3×3Conv dilation=2), a batch normalization layer (BatchNorm), and a nonlinear activation layer (ReLU). The second network module may include a 1×1 convolution layer, a batch normalization layer, and a nonlinear activation layer. The third network module may include a 3×3 convolution layer, a batch normalization layer, and a nonlinear activation layer. Taking the red pixel channel in the Bayer image as an example, the 3×3 convolution kernel with a convolution expansion coefficient of 2 can be as follows: Figure 4 As shown. Among them, Figure 4 Here, R represents a red pixel, B represents a blue pixel, Gr represents a green pixel adjacent to the red pixel, and Gb represents a green pixel adjacent to the blue pixel.
[0057] In one embodiment, Figure 5 As shown, the cross-channel convolution module may include a second input feature module, a fourth network module, a second network module, a first network module, a second network module and a second output feature module connected in sequence.
[0058] In the cross-channel convolution module, the fourth network module may include a 3×3 locally connected convolution layer (3×3LConv) with a convolution expansion factor of 1, a batch normalization layer, and a nonlinear activation layer. The 3×3 locally connected convolution layer can apply different convolution parameters to different channels in the Bayer image. Taking the Bayer image as an example, the 3×3 cross-channel convolution kernel with a convolution expansion factor of 1 can be as follows: Figure 4 shown.
[0059] In one embodiment, Figure 6 As shown, the residual network module may include a third input feature module, a second network module, a fifth network module, an adder module, a nonlinear activation module, and a third output feature module connected in sequence. Among them, the fifth network module includes a 3×3 convolution layer (3×3Conv) and a batch normalization layer. The third input feature module is also connected to the adder module.
[0060] It should be noted that, in the embodiment of the present application, the first input feature module, the second input feature module, and the third input feature module are modules for receiving feature maps of images or for processing feature maps. The first output feature module, the second output feature module, and the third output feature module are modules for outputting feature maps of images that have undergone various processing.
[0061] See also Figure 7 , Figure 7 A second flow chart of the image processing method provided in the embodiment of the present application may include:
[0062] 201. Acquire a first image.
[0063] For example, the electronic device may first acquire an image, which may be recorded as a first image. It is understandable that the first image is an image that needs to be detected and corrected for bad pixels.
[0064] In one embodiment, the first image may be a Bayer image.
[0065] 202. Perform same-channel convolution processing on the first image to obtain a second image, and perform cross-channel convolution processing on the first image to obtain a third image, wherein the convolution expansion coefficient corresponding to the same-channel convolution processing is 2, and the convolution expansion coefficient corresponding to the cross-channel convolution processing is 1.
[0066] For example, after acquiring the first image, the electronic device may perform same-channel convolution processing on the first image to obtain a second image, wherein the convolution expansion coefficient corresponding to the same-channel convolution processing is 2. Furthermore, the electronic device may also perform cross-channel convolution processing on the first image to obtain a third image, wherein the convolution expansion coefficient corresponding to the cross-channel convolution processing is 1.
[0067] In the embodiments of this application, please refer to Figure 4 , Figure 4 The convolution kernel corresponding to the same-channel convolution processing with a convolution expansion coefficient of 2 and the convolution kernel corresponding to the cross-channel convolution processing with a convolution expansion coefficient of 1 are shown.
[0068] 203. Perform feature stitching on the second image and the third image to obtain a fourth image.
[0069] 204. Process the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel.
[0070] 205. When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel.
[0071] For example, 203, 204, and 205 may include:
[0072] After obtaining the second image and the third image, the electronic device may perform feature stitching on the second image and the third image to obtain a fourth image.
[0073] After obtaining the fourth image, the electronic device may process the fourth image using at least one residual network to determine the probability value of whether the pixel in the first image is a bad pixel. For example, when the fourth image is processed using the residual network, the probability value of whether the pixel is a bad pixel may be output for each pixel in the first image. When the probability value is greater than a preset threshold, the electronic device may detect the corresponding pixel as a bad pixel.
[0074] In the embodiment of the present application, the convolution perception field is used to determine the probability value of whether a pixel is a bad pixel, wherein the number of rows and columns of the convolution perception field are both N p , N p =7+2N r , where N r is the number of residual networks used in the process of 204. For example, if the electronic device uses one residual network to process the fourth image, then the number of rows and columns of the convolution receptive field are both 9, that is, the window size of the convolution receptive field is 9×9. For another example, if the electronic device uses two residual networks to process the fourth image, then the number of rows and columns of the convolution receptive field are both 11, that is, the window size of the convolution receptive field is 11×11.
[0075] It is understandable that in the embodiment of the present application, the more residual networks are used, the stronger the bad pixel image block detection capability will be, where the bad pixel image block refers to the image block composed of bad pixel pixels. For example, when two residual networks are used, the solution has the ability to detect bad pixel image blocks of size 2×2. When four residual networks are used, the solution has the ability to detect bad pixel image blocks of size 3×3, and so on.
[0076] In practical applications, the specific number of residual networks to be used can be determined based on the actual computing power and cache size of the electronic device. For example, when the computing power and cache of the electronic device are both small, a smaller number of residual networks can be used, and when the computing power and cache of the electronic device are both large, a larger number of residual networks can be used. When more residual networks are used, it is equivalent to using a larger convolutional receptive field, so the electronic device will use more line buffers, so that the solution has the ability to detect larger bad pixel image blocks.
[0077] 206. Perform pixel value correction of the bad pixel by non-local filtering, wherein, in the process of performing the pixel value correction of the non-local filtering, when calculating the correction weight of the center pixel of the neighborhood image block within the search range for the bad pixel currently being corrected, the bad pixel detected in the current image block and the neighborhood image block is excluded from the calculation process, and when calculating the correction weight, the number of valid matching pixel pairs of the neighborhood image block and the current image block is added as a calculation factor to the calculation of the correction weight, the valid matching pixel pair is non-bad pixel pixels that match each other, and the current image block is an image block with the bad pixel currently being corrected as the center pixel.
[0078] For example, after detecting bad pixel from the first image, the electronic device can perform pixel value correction of the bad pixel by non-local filtering, wherein, in the process of performing the pixel value correction of the non-local filtering, when calculating the correction weight of the central pixel of the neighborhood image block within the search range for the bad pixel currently being corrected, the bad pixel detected in the current image block and the neighborhood image block is excluded from the calculation process, and when calculating the correction weight, the number of valid matching pixel pairs of the neighborhood image block and the current image block is added as a calculation factor to the calculation of the correction weight, and the valid matching pixel pair is a non-bad pixel that matches each other. The current image block is an image block with the bad pixel currently being corrected as the central pixel.
[0079] In the embodiment of the present application, the formula used to correct the first image may be as follows:
[0080]
[0081] in,
[0082]
[0083] e(N i ,N j )=|min(d(N i ),d(N j ))≤T prob |
[0084] Among them, v(i) is the pixel value of pixel i in the first image, and NL[v](i) is the pixel value of the pixel corresponding to the pixel i in the corrected first image.
[0085] For example, if the pixel in the first row and first column of the first image is recorded as v(1), then the pixel in the first row and first column of the corrected first image is recorded as NL[v](1). For another example, if the pixel in the first row and second column of the first image is recorded as v(2), then the pixel in the first row and second column of the corrected first image is recorded as NL[v](2), and so on.
[0086] d(i) is the probability that pixel i is a bad pixel. prob It is the threshold value for judging whether a pixel is a bad pixel. When d(i)>T prob When d(i)≤T prob When the pixel i is judged as a non-defective pixel.
[0087] For example, d(1) can be used to represent the probability that the pixel in the first row and first column of the first image is a bad pixel. prob , then it can be determined that the pixel in the first row and first column of the first image is a bad pixel. If d(1) is less than or equal to T prob , then it can be determined that the pixel in the first row and first column in the first image is not a bad pixel.
[0088] I is the search range of non-local filtering for the current pixel i, v(j) is the pixel value of the central pixel j of the neighborhood image block within the search range corresponding to pixel i, w(i,j) is the correction weight of pixel j to pixel i, d(j) is the probability that pixel j is a bad pixel, when d(j)>T prob When d(j)≤T prob When the pixel j is judged as a non-defective pixel.
[0089] For example, the search range of the non-local filtering of the current pixel i is 8×8, then I is the range of the 8×8 image block. For example, the size of the current image block with pixel i as the center pixel is 5×5, then the size of the neighborhood image block is also 5×5, the center pixel of the neighborhood image block is pixel j, and w(i,j) is the correction weight of pixel j to pixel i. In some embodiments, the electronic device can calculate the Euclidean distance between the current image block and the neighborhood image block. If the Euclidean distance between the two image blocks is smaller, it means that the two image blocks are more similar, then the correction weight of the center pixel j of the neighborhood image block to pixel i can be larger. If the Euclidean distance between the two image blocks is larger, it means that the two image blocks are less similar, then the correction weight of the center pixel j of the neighborhood image block to pixel i can be smaller.
[0090] Z(i) is a normalization term, ε is a constant parameter, N is the number of valid matching pixel pairs between the neighborhood image block and the current image block, and k is a filtering parameter.
[0091] For example, the sizes of the neighborhood image block and the current image block are both 5×5 (i.e., a total of 25 pixels). If the pixels contained in the neighborhood image block and the current image block are not all non-bad pixel pixels, for example, the pixels in the first row and first column of the neighborhood image block are bad pixel pixels, and the pixels in the first row and first column of the current image block are also bad pixel pixels, that is, there is one pixel pair between the neighborhood image block and the current image block that cannot be effectively matched, and the remaining pixels are all non-bad pixel pixels, so the number N of valid matching pixel pairs between the neighborhood image block and the current image block is 24 (i.e., 25-1). For another example, the pixels in the first row and first column of the neighborhood image block are bad pixel pixels, and the pixels in the first row and second column of the current image block are bad pixel pixels, that is, there are two pixel pairs between the neighborhood image block and the current image block that cannot be effectively matched, and the remaining pixels are all non-bad pixel pixels, so the number N of valid matching pixel pairs between the neighborhood image block and the current image block is 23 (i.e., 25-2), and so on.
[0092] N i Indicates the pixels contained in the current image block where pixel i is located, N j represents the pixels contained in the neighborhood image block corresponding to pixel j, v(N i )-v(N j ) represents the matrix obtained by subtracting the pixel values of each matching pixel in the current image block from the pixel values in the neighborhood image block.
[0093] For example, if the pixel value of the first row and first column of the current image block is 109, and the pixel value of the first row and first column of the neighboring image block is 108, then their difference is 1, then v(N i )-v(N j) represents the value of the first row and first column of the matrix represented by 1. For another example, the pixel value of the first row and second column of the current image block is 109, and the pixel value of the first row and second column of the neighboring image block is 106, then their difference is 3, then v(N i )-v(N j ) is represented by the matrix with a value of 3 at the first row and second column.
[0094] d(N i ) represents the probability that the pixels contained in the current image block are bad pixels, d(N j ) represents the probability that the pixels contained in the neighborhood image block are bad pixels, e(N i , N j ) represents the mask matrix of the valid matching pixels of the current image block and the neighborhood image block corresponding to pixel i, wherein when the matching pixels of the current image block and the neighborhood image block are both non-defective pixels, the value at the corresponding position of the mask matrix is 1, otherwise it is 0.
[0095] For example, if the pixels in the first row and first column of the current image block are good pixels, and the pixels in the first row and first column of the neighboring image block are also good pixels, then the value at the first row and first column of the mask matrix is 1. For another example, if the pixels in the first row and second column of the current image block are good pixels, and the pixels in the first row and second column of the neighboring image block are bad pixels, then the value at the first row and second column of the mask matrix is 0. For another example, if the pixels in the first row and third column of the current image block are bad pixels, and the pixels in the first row and third column of the neighboring image block are also bad pixels, then the value at the first row and third column of the mask matrix is 0.
[0096] Therefore, in an embodiment of the present application, if pixel i in the first image is not a bad pixel, then the pixel value NL[v](i) of the pixel corresponding to the pixel i in the corrected second image is equal to the pixel value v(i) of the pixel i in the first image.
[0097] If the pixel i in the first image is a bad pixel, the pixel value NL[v](i) of the pixel corresponding to the pixel i in the corrected image is calculated according to the following formula:
[0098]
[0099] It can be understood that the non-local filtering module provided in this embodiment is based on the classic non-local filtering scheme. At the same time, due to the existence of bad pixels whose true pixel values are unknown, the weighting rule is modified in the embodiment of the present application, and the bad pixels are excluded when calculating the correction weight of the central pixel j of the neighborhood image block to the central pixel i of the current image block. At the same time, the number N of valid matching pixel pairs of the neighborhood image block and the current image block is added as a calculation factor to the calculation of the weight. Finally, based on the normalized similarity of block matching and the number N of valid matching pixel pairs, the prediction weight of the corresponding matching pixel j when predicting the central bad pixel i can be determined, and the final recovery value can be estimated accordingly.
[0100] In other implementations, this embodiment may also dynamically select the size of the image block (the current image block and the neighborhood image block) and the size of the search range I according to the amount of calculation and the size of the cache.
[0101] In one embodiment, the process from 202 to 206 can be completed by a pre-trained image restoration model. The pre-trained image restoration model can be a network model that has been trained. The pre-trained image restoration model can be used to detect and correct bad pixels in an image. Therefore, after acquiring the first image and the pre-trained image restoration model, the electronic device can input the first image into the pre-trained image restoration model, and the pre-trained image restoration model can detect and correct bad pixels in the first image, and output the corrected image.
[0102] The pre-trained image restoration model may include at least a same-channel convolution module, a cross-channel convolution module, a residual network module, and a non-local filtering module. The same-channel convolution module and the cross-channel convolution module may be connected in parallel and then connected to the residual network module. The residual network module may be connected to the non-local filtering module, such as Figure 8 shown.
[0103] In an embodiment of the present application, the same-channel convolution module, the cross-channel convolution module and the residual network module in the trained model can be used to detect bad pixels. Among them, the same-channel convolution module is used to perform same-channel convolution processing on the first image to obtain the second image, the cross-channel convolution module is used to perform cross-channel convolution processing on the first image to obtain the third image, and the residual network module is used to process the fourth image using at least one residual network to determine the probability value of whether the pixel in the first image is a bad pixel. When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel. The non-local filtering correction module can be used to correct the detected bad pixels.
[0104] In an embodiment of the present application, the module for correcting the detected bad pixel may be a non-local filtering module. In the process of performing non-local filtering pixel correction on the detected bad pixel, when calculating the correction weight of the central pixel of the neighborhood image block within the search range for the bad pixel currently being corrected, the bad pixel detected in the current image block and the neighborhood image block may be excluded from the weight calculation process, wherein the current image block is an image block with the bad pixel currently being corrected as the central pixel. Furthermore, when calculating the above correction weight, the non-local filtering module may add the number of valid matching pixel pairs of the neighborhood image block and the current image block as a calculation factor to the calculation of the above correction weight, wherein the valid matching pixel pair may refer to non-bad pixel pixels that match each other.
[0105] See also Fig. 9 , Fig. 9 A third flow chart of the image processing method provided in the embodiment of the present application, the flow chart may include:
[0106] 301. Acquire a first image.
[0107] For example, the electronic device may first acquire an image, which may be recorded as a first image. It is understandable that the first image is an image that needs to be detected and corrected for bad pixels.
[0108] In one embodiment, the first image may be a Bayer image.
[0109] 302. Perform same-channel convolution processing on the first image to obtain a second image, and perform cross-channel convolution processing on the first image to obtain a third image, wherein the convolution expansion coefficient corresponding to the same-channel convolution processing is 2, and the convolution expansion coefficient corresponding to the cross-channel convolution processing is 1.
[0110] For example, after acquiring the first image, the electronic device may perform same-channel convolution processing on the first image to obtain a second image, wherein the convolution expansion coefficient corresponding to the same-channel convolution processing is 2. Furthermore, the electronic device may also perform cross-channel convolution processing on the first image to obtain a third image, wherein the convolution expansion coefficient corresponding to the cross-channel convolution processing is 1.
[0111] In the embodiments of this application, please refer to Figure 4 , Figure 4 The convolution kernel corresponding to the same-channel convolution processing with a convolution expansion coefficient of 2 and the convolution kernel corresponding to the cross-channel convolution processing with a convolution expansion coefficient of 1 are shown.
[0112] 303. Perform feature stitching on the second image and the third image to obtain a fourth image.
[0113] 304. Process the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel.
[0114] 305. When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel.
[0115] For example, 303, 304, and 305 may include:
[0116] After obtaining the second image and the third image, the electronic device may perform feature stitching on the second image and the third image to obtain a fourth image.
[0117] After obtaining the fourth image, the electronic device may process the fourth image using at least one residual network to determine the probability value of whether the pixel in the first image is a bad pixel. For example, when the fourth image is processed using the residual network, the probability value of whether the pixel is a bad pixel may be output for each pixel in the first image. When the probability value is greater than a preset threshold, the electronic device may detect the corresponding pixel as a bad pixel.
[0118] In the embodiment of the present application, the convolution perception field is used to determine the probability value of whether a pixel is a bad pixel, wherein the number of rows and columns of the convolution perception field are both N p , N p =7+2N r , where N r is the number of residual networks used in the process of 304. For example, if the electronic device uses one residual network to process the fourth image, the number of rows and columns of the convolution receptive field are both 9, that is, the window size of the convolution receptive field is 9×9. For another example, if the electronic device uses two residual networks to process the fourth image, the number of rows and columns of the convolution receptive field are both 11, that is, the window size of the convolution receptive field is 11×11.
[0119] It is understandable that in the embodiment of the present application, the more residual networks are used, the stronger the bad pixel image block detection capability will be, where the bad pixel image block refers to the image block composed of bad pixel pixels. For example, when two residual networks are used, the solution has the ability to detect bad pixel image blocks of size 2×2. When four residual networks are used, the solution has the ability to detect bad pixel image blocks of size 3×3, and so on.
[0120] In practical applications, the specific number of residual networks to be used can be determined based on the actual computing power and cache size of the electronic device. For example, when the computing power and cache of the electronic device are both small, a smaller number of residual networks can be used, and when the computing power and cache of the electronic device are both large, a larger number of residual networks can be used. When more residual networks are used, it is equivalent to using a larger convolutional receptive field, so the electronic device will use more line buffers, so that the solution has the ability to detect larger bad pixel image blocks.
[0121] 306. Calculate the weights of various interpolation strategies used by the current region for the bad pixel currently being corrected, where the current region is a region with a preset size with the bad pixel currently being corrected as the center pixel, and each interpolation strategy corresponds to an interpolation type.
[0122] 307. Calculate the correction value of the bad pixel currently being corrected according to the weights of various interpolation strategies, and perform correction according to the correction value.
[0123] For example, after detecting the bad pixel in the first image, the electronic device can correct each bad pixel. In this embodiment, when correcting each bad pixel, the electronic device can determine a region with a preset size with the bad pixel as the center pixel as the current region, and calculate the weight of the current region using various interpolation strategies for the center bad pixel, wherein each interpolation strategy corresponds to an interpolation type. Afterwards, the electronic device can calculate the correction value of the center bad pixel according to the weights of using various interpolation strategies, and use the correction value to correct the bad pixel.
[0124] It can be understood that the sum of the weights of various interpolation strategies is 1.
[0125] It should be noted that, in this embodiment, the interpolation types (interpolation strategies) may include: horizontal interpolation (H), horizontal interpolation based on the current peak value (H+peak), horizontal interpolation based on the current valley value (H+valley), vertical interpolation (V), vertical interpolation based on the current peak value (V+peak), vertical interpolation based on the current valley value (V+valley), 45-degree oblique interpolation (D45), 45-degree oblique interpolation based on the current peak value (D45+peak), 45-degree oblique interpolation based on the current valley value (D45+valley), 135-degree oblique interpolation (D135), 135-degree oblique interpolation based on the current peak value (D135+peak), 135-degree oblique interpolation based on the current valley value (D135+valley), average interpolation in each direction (Flat), and nearest neighbor interpolation (Texture).
[0126] When calculating the correction value of a bad pixel, the electronic device can first calculate the interpolation values corresponding to various interpolation strategies, then perform weighted summation on all the interpolation values according to the weights corresponding to the various interpolation strategies, and determine the weighted sum as the correction value corresponding to the bad pixel.
[0127] In one embodiment, the process from 302 to 307 can be completed by a pre-trained image restoration module. The pre-trained image restoration model can be used to detect and correct bad pixel pixels in an image. The pre-trained image restoration model can at least include a same-channel convolution module, a cross-channel convolution module, a residual network module, and a deep learning bad pixel restoration module, such as Fig.10 As shown. The same-channel convolution module and the cross-channel convolution module are connected in parallel to the residual network module, and the residual network module is connected to the deep learning bad pixel recovery module. The deep learning bad pixel recovery module can be used to calculate the weights of various interpolation strategies for the central bad pixel pixels in the current area, and obtain the correction value of the central bad pixel according to the weight calculation using various interpolation strategies, and correct the bad pixel according to the correction value, and each interpolation strategy corresponds to an interpolation type.
[0128] Therefore, the electronic device can input the first image into the pre-trained image restoration model, and the pre-trained image restoration model detects and corrects the bad pixel of the first image, and outputs the corrected image. The pre-trained image restoration model can first use its same-channel convolution module, cross-channel convolution module and residual network model to detect the bad pixel in the first image. Then, the pre-trained image restoration module can use its deep learning bad pixel recovery module to correct each bad pixel. Among them, when correcting each bad pixel, the deep learning bad pixel recovery module can determine an area with the bad pixel as the center pixel as the current area, and calculate the weight of the current area using various interpolation strategies for the center bad pixel, and each interpolation strategy corresponds to an interpolation type. Afterwards, the pre-trained image restoration module can calculate the correction value of the center bad pixel according to the weights of various interpolation strategies, and use the correction value to correct the bad pixel.
[0129] In the embodiment of the present application, the deep learning bad pixel recovery module is a trained deep learning based module. The deep learning bad pixel recovery module may at least include: a fourth input feature module, a sixth network module, a calculation module and a recovery module, such as Fig.11As shown. The sixth network module may include a 1×1 convolution layer and a softmax classification layer. The sixth network module is used to calculate the weights of various interpolation strategies used by the current region for the central bad pixel. The calculation module is used to calculate the correction value of the central bad pixel according to the weights of various interpolation strategies. The recovery module is used to correct the bad pixel according to the calculated correction value.
[0130] It should be noted that, in this embodiment, Fig.11 As shown, the interpolation types (interpolation strategies) corresponding to the sixth network module may include: horizontal interpolation (H), horizontal interpolation based on the current peak value (H+peak), horizontal interpolation based on the current valley bottom (H+valley), vertical interpolation (V), vertical interpolation based on the current peak value (V+peak), vertical interpolation based on the current valley bottom (V+valley), 45-degree oblique interpolation (D45), 45-degree oblique interpolation based on the current peak value (D45+peak), 45-degree oblique interpolation based on the current valley bottom (D45+valley), 135-degree oblique interpolation (D135), 135-degree oblique interpolation based on the current peak value (D135+peak), 135-degree oblique interpolation based on the current valley bottom (D135+valley), average interpolation in each direction (Flat), and nearest neighbor interpolation (Texture).
[0131] The calculation module can be used to calculate the correction value of the bad pixel. When calculating, the calculation module can first calculate the interpolation values corresponding to various interpolation strategies, and then perform weighted summation on all the interpolation values according to the weights corresponding to the various interpolation strategies, and the weighted sum value can be determined as the correction value corresponding to the bad pixel.
[0132] In one embodiment, the bad pixel correction module in the pre-trained image restoration model may also include a non-local filtering module and a deep learning bad pixel recovery module. Based on this, for each bad pixel, the corresponding correction value can be calculated using the non-local filtering module and the deep learning bad pixel recovery module respectively. Then, the electronic device can obtain the first weight corresponding to the non-local filtering module and the second weight corresponding to the deep learning bad pixel recovery module. Afterwards, for each bad pixel, the electronic device can perform weighted summation based on the correction value calculated by the non-local filtering module and its corresponding first weight, and the correction value calculated by the deep learning bad pixel recovery module and its corresponding second weight, and determine the weighted sum as the final correction value of the bad pixel.
[0133] The following is an explanation of the training process of the pre-trained image restoration model. Fig.12As shown, the training model may include a same-channel convolution module, a cross-channel convolution module, and a residual network module. The structural schematic diagrams of the same-channel convolution module, the cross-channel convolution module, and the residual network module can be respectively as shown in Figure 3 , 5 , as shown in Figure 6.
[0134] During learning and training, the data of each training sample may include image data and a mask of bad pixel pixels. During learning and training, a sigmoid function may be used as an activation function. The loss function may apply the fused TverskyLoss function and Focal-loss function as the loss function for model training. Among them, the fused Tversky Loss function and Focal-loss function may refer to the weighted value of the Tversky Loss function value and the Focal-loss function value as the value of the loss function during model training. The weight values corresponding to the Tversky Loss function and the Focal-loss function may be set as needed. For example, the weight values corresponding to the Tversky Loss function and the Focal-loss function may both be 50%, and so on.
[0135] In one embodiment, the image data in the training sample may not be the data of a complete image, but the data of a local image in an image. In view of the imbalance between bad pixel pixels and non-bad pixel pixels, using the fused Tversky Loss function and Focal-loss function as the loss function for model training is beneficial to model training.
[0136] When the bad pixel correction module includes a deep learning bad pixel recovery module, the structural diagram of the training model can be as follows: Fig.13 As shown, the training model can include a same-channel convolution module, a cross-channel convolution module, a residual network module, and a deep learning bad pixel recovery module.
[0137] It can be understood that the embodiment of the present application provides a flexible bad pixel recovery method, which can control the actual correction capability and accuracy of the bad pixel detection scheme by flexibly adjusting the size of the convolution receptive field and the number of convolution channels according to the acceptable cache area and computing power in actual needs.
[0138] Since the pre-trained image restoration model in the embodiment of the present application is trained by deep learning, the embodiment of the present application can obtain a higher bad pixel classification accuracy rate compared with the traditional solution under the same convolution perception field size. In addition, the embodiment of the present application can effectively improve the bad pixel recovery ability of multi-bad pixel image blocks by adding more residual network modules. At the same time, due to the improvement of bad pixel classification accuracy and the improved bad pixel recovery method, the boundary or texture degradation problem that may be caused by bad pixel correction can be avoided.
[0139] See also Fig.14 , Fig.14 The image processing device 400 may include an acquisition module 401 and a processing module 402 .
[0140] An acquisition module 401 is used to acquire a first image;
[0141] The processing module 402 is used to perform same-channel convolution processing on the first image to obtain a second image, and to perform cross-channel convolution processing on the first image to obtain a third image; to perform feature stitching on the second image and the third image to obtain a fourth image; to process the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel; when the probability value is greater than a preset threshold, the pixel is detected as a bad pixel; and to correct the bad pixel.
[0142] In one implementation, the convolution expansion coefficient corresponding to the same-channel convolution processing is 2.
[0143] In one embodiment, the convolution expansion coefficient corresponding to the cross-channel convolution processing is 1.
[0144] In one embodiment, the convolution receptive field is used to determine the probability value of whether a pixel is a bad pixel, wherein the number of rows and columns of the convolution receptive field are both N. p , N p =7+2N r , where N r is the number of at least one residual network.
[0145] In one embodiment, the processing module 402 is used to perform pixel value correction of bad pixel by non-local filtering, wherein, in the process of performing pixel value correction of non-local filtering, when calculating the correction weight of the center pixel of the neighborhood image block within the search range for the bad pixel currently being corrected, the bad pixel detected in the current image block and the neighborhood image block is excluded from the calculation process, and the current image block is an image block with the bad pixel currently being corrected as the center pixel.
[0146] In one implementation, when calculating the correction weight, the number of valid matching pixel pairs between the neighborhood image block and the current image block is added as a calculation factor into the calculation of the correction weight, and the valid matching pixel pairs are mutually matching non-bad pixel pixels.
[0147] In one implementation, the formula used when correcting the first image is as follows:
[0148]
[0149] in,
[0150]
[0151] e(N i ,N j )=|min(d(N i ),d(N j ))≤T prob |
[0152] Wherein, v(i) is the pixel value of pixel i in the first image, and NL[v](i) is the pixel value of the pixel corresponding to the pixel i in the corrected image;
[0153] d(i) is the probability that the pixel i is a bad pixel;
[0154] T prob It is the threshold value for judging whether a pixel is a bad pixel. When d(i)>T prob When d(i)≤T prob When the pixel i is judged to be a non-defective pixel;
[0155] I is the search range of the non-local filtering of the current pixel i, v(j) is the pixel value of the central pixel j of the neighborhood image block within the search range corresponding to the pixel i, w(i,j) is the correction weight of pixel j to pixel i, d(j) is the probability that the pixel j is a bad pixel, when d(j)>T prob When d(j)≤T prob It is determined that the pixel j is a non-defective pixel;
[0156] Z(i) is a normalization term, ε is a constant parameter, N is the number of valid matching pixel pairs between the neighborhood image block and the current image block, and k is a filtering parameter;
[0157] N i represents the pixels contained in the current image block where the pixel i is located, N j represents the pixels contained in the neighborhood image block corresponding to the pixel i, v(N i )-v(N j ) represents a matrix obtained by subtracting the pixel values of each matching pixel in the current image block from the pixel values in the neighborhood image block;
[0158] d(N i ) represents the probability that the pixel contained in the current image block is a bad pixel, d(N j ) represents the probability that the pixels contained in the neighborhood image block are bad pixels, e(N i , N j) represents the mask matrix of the valid matching pixels of the current image block and the neighborhood image block corresponding to the pixel i, wherein when the matching pixels of the current image block and the neighborhood image block are both non-dead pixels, the value at the corresponding position of the mask matrix is 1, otherwise it is 0.
[0159] In one embodiment, the processing module 402 is used to: calculate the weights of various interpolation strategies used in the current area for the bad pixel currently being corrected, the current area is an area of a preset size with the bad pixel currently being corrected as the center pixel, and each interpolation strategy corresponds to an interpolation type; calculate the correction value of the bad pixel currently being corrected based on the weights of the various interpolation strategies used, and perform correction based on the correction value.
[0160] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the process in the image processing method provided in this embodiment.
[0161] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the processor is used to execute the process in the image processing method provided in the present embodiment by calling a computer program stored in the memory.
[0162] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Fig.15 , Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0163] The electronic device 500 may include components such as a sensor 501, a memory 502, and a processor 503. Those skilled in the art will appreciate that Fig.15 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0164] The sensor 501 may include a gyro sensor (eg, a three-axis gyro sensor), an acceleration sensor, or other sensors.
[0165] The memory 502 may be used to store applications and data. The applications stored in the memory 502 include executable codes. The applications may be composed of various functional modules. The processor 503 executes various functional applications and data processing by running the applications stored in the memory 502.
[0166] The processor 503 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the electronic device as a whole.
[0167] In this embodiment, the processor 503 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 503 will run the application program stored in the memory 502 to execute:
[0168] acquiring a first image;
[0169] Performing same-channel convolution processing on the first image to obtain a second image, and performing cross-channel convolution processing on the first image to obtain a third image;
[0170] Performing feature stitching on the second image and the third image to obtain a fourth image;
[0171] Processing the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel;
[0172] When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel;
[0173] Corrects dead pixels.
[0174] See also Fig.16 The electronic device 500 may include components such as a sensor 501, a memory 502, a processor 503, a battery 504, a display screen 505, a speaker 506, and a microphone 507.
[0175] The sensor 501 may include a gyro sensor (eg, a three-axis gyro sensor), an acceleration sensor, or other sensors.
[0176] The memory 502 may be used to store applications and data. The applications stored in the memory 502 include executable codes. The applications may be composed of various functional modules. The processor 503 executes various functional applications and data processing by running the applications stored in the memory 502.
[0177] The processor 503 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the electronic device as a whole.
[0178] The battery 504 can be used to provide power support for various components and modules of the electronic device, thereby ensuring the normal operation of various components and modules.
[0179] The display screen 505 can be used to display information such as text, images, etc.
[0180] The microphone 506 can be used to collect sound signals in the surrounding environment, such as the voice uttered by the user.
[0181] In this embodiment, the processor 503 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 503 will run the application program stored in the memory 502 to execute:
[0182] acquiring a first image;
[0183] Performing same-channel convolution processing on the first image to obtain a second image, and performing cross-channel convolution processing on the first image to obtain a third image;
[0184] Performing feature stitching on the second image and the third image to obtain a fourth image;
[0185] Processing the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel;
[0186] When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel;
[0187] Corrects dead pixels.
[0188] In one implementation, the convolution expansion coefficient corresponding to the same-channel convolution processing is 2.
[0189] In one embodiment, the convolution expansion coefficient corresponding to the cross-channel convolution processing is 1.
[0190] In one embodiment, the convolution receptive field is used to determine the probability value of whether a pixel is a bad pixel, wherein the number of rows and columns of the convolution receptive field are both N. p , N p =7+2N r , where N r is the number of at least one residual network.
[0191] In one embodiment, when the processor 502 performs correction for bad pixel, it can perform: performing pixel value correction for the bad pixel by non-local filtering, wherein, in the process of performing pixel value correction by non-local filtering, when calculating the correction weight of the center pixel of the neighborhood image block within the search range for the bad pixel currently being corrected, the bad pixel detected in the current image block and the neighborhood image block is excluded from the calculation process, and the current image block is an image block with the bad pixel currently being corrected as the center pixel.
[0192] In one implementation, when calculating the correction weight, the number of valid matching pixel pairs between the neighborhood image block and the current image block is added as a calculation factor into the calculation of the correction weight, and the valid matching pixel pairs are mutually matching non-bad pixel pixels.
[0193] In one implementation, the formula used to correct the first image is as follows:
[0194]
[0195] in,
[0196]
[0197] e(N i ,N j )=|min(d(N i ),d(N j ))≤T prob |
[0198] Wherein, v(i) is the pixel value of pixel i in the first image, and NL[v](i) is the pixel value of the pixel corresponding to the pixel i in the corrected image;
[0199] d(i) is the probability that the pixel i is a bad pixel;
[0200] T prob It is the threshold value for judging whether a pixel is a bad pixel. When d(i)>T prob When d(i)≤T prob When the pixel i is judged to be a non-defective pixel;
[0201] I is the search range of the non-local filtering of the current pixel i, v(j) is the pixel value of the central pixel j of the neighborhood image block within the search range corresponding to the pixel i, w(i,j) is the correction weight of pixel j to pixel i, d(j) is the probability that the pixel j is a bad pixel, when d(j)>T prob When d(j)≤T prob It is determined that the pixel j is a non-defective pixel;
[0202] Z(i) is a normalization term, ε is a constant parameter, N is the number of valid matching pixel pairs between the neighborhood image block and the current image block, and k is a filtering parameter;
[0203] N i represents the pixels contained in the current image block where the pixel i is located, N j represents the pixels contained in the neighborhood image block corresponding to the pixel i, v(N i )-v(N j ) represents a matrix obtained by subtracting the pixel values of each matching pixel in the current image block from the pixel values in the neighborhood image block;
[0204] d(N i ) represents the probability that the pixel contained in the current image block is a bad pixel, d(N j ) represents the probability that the pixels contained in the neighborhood image block are bad pixels, e(N i , N j ) represents the mask matrix of the valid matching pixels of the current image block and the neighborhood image block corresponding to the pixel i, wherein when the matching pixels of the current image block and the neighborhood image block are both non-dead pixels, the value at the corresponding position of the mask matrix is 1, otherwise it is 0.
[0205] In one embodiment, when the processor 503 performs correction on bad pixel, it can execute: calculating the weights of various interpolation strategies used in the current area for the bad pixel currently being corrected, the current area being an area of a preset size with the bad pixel currently being corrected as the center pixel, and each interpolation strategy corresponds to an interpolation type; calculating the correction value of the bad pixel currently being corrected based on the weights of various interpolation strategies, and performing correction based on the correction value.
[0206] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the image processing method above, and will not be repeated here.
[0207] The image processing device provided in the embodiment of the present application belongs to the same concept as the image processing method in the above embodiment. Any method provided in the image processing method embodiment can be run on the image processing device. The specific implementation process is detailed in the image processing method embodiment and will not be repeated here.
[0208] It should be noted that, for the image processing method described in the embodiment of the present application, a person of ordinary skill in the art can understand that the implementation of all or part of the process of the image processing method described in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the image processing method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.
[0209] For the image processing device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.
[0210] The above is a detailed introduction to an image processing method, device, storage medium and electronic device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. An image processing method, characterized in that: include: acquiring a first image; The same-channel convolution module is used to perform same-channel convolution processing on the first image to obtain a second image, and the cross-channel convolution module is used to perform cross-channel convolution processing on the first image to obtain a third image; wherein the same-channel convolution module includes a first input feature module, a first network module, a second network module, a third network module, a second network module and a first output feature module connected in sequence; the cross-channel convolution module includes a second input feature module, a fourth network module, a second network module, a first network module, a second network module and a second output feature module connected in sequence; the convolution expansion coefficient of the first network module is 2, and the convolution expansion coefficient of the fourth network module is 1; Performing feature stitching on the second image and the third image to obtain a fourth image; Processing the fourth image using at least one residual network to determine a probability value of whether a pixel in the first image is a bad pixel; When the probability value is greater than a preset threshold, the pixel is detected as a bad pixel; Correct the bad pixel.
2. The image processing method according to claim 1, characterized in that: The convolution receptive field is used to determine the probability value of whether the pixel is the bad pixel, wherein the number of rows and columns of the convolution receptive field are both N. p , N p =7+2N r , where N r is the number of the at least one residual network.
3. The image processing method according to claim 1, characterized in that: Correcting the bad pixel includes: A pixel value correction of the bad pixel is performed by non-local filtering, wherein, in the process of performing the pixel value correction by non-local filtering, when calculating the correction weight of the central pixel of the neighborhood image block within the search range for the bad pixel currently being corrected, the bad pixel detected in the current image block and the neighborhood image block is excluded from the calculation process, and the current image block is an image block with the bad pixel currently being corrected as the central pixel.
4. The image processing method according to claim 3, characterized in that: When calculating the correction weight, the number of valid matching pixel pairs between the neighborhood image block and the current image block is added as a calculation factor into the calculation of the correction weight, and the valid matching pixel pairs are non-bad pixel pairs that match each other.
5. The image processing method according to claim 4, characterized in that: The formula used to correct the first image is as follows: in, e(N i ,N j )=|min(d(N i ),d(N j ))≤T prob | Wherein, v(i) is the pixel value of pixel i in the first image, and NL[v](i) is the pixel value of the pixel corresponding to the pixel i in the corrected image; d(i) is the probability that the pixel i is a bad pixel; T prob It is the threshold value for judging whether a pixel is a bad pixel. When d(i)>T prob When d(i)≤T prob When the pixel i is judged to be a non-defective pixel; I is the search range of non-local filtering of the pixel i, v(j) is the pixel value of the central pixel j of the neighborhood image block within the search range corresponding to the pixel i, w(i,j) is the correction weight of pixel j to pixel i, d(j) is the probability that the pixel j is a bad pixel, when d(j)>T prob When d(j)≤T prob It is determined that the pixel j is a non-defective pixel; Z(i) is a normalization term, ε is a constant parameter, N is the number of valid matching pixel pairs between the neighborhood image block and the current image block, and k is a filtering parameter; N i represents the pixels contained in the current image block where the pixel i is located, N j represents the pixels contained in the neighborhood image block corresponding to the pixel i, v(N i )-v(N j ) represents a matrix obtained by subtracting the pixel values of each matching pixel in the current image block from the pixel values in the neighborhood image block; d(N i ) represents the probability that the pixel contained in the current image block is a bad pixel, d(N j ) represents the probability that the pixels contained in the neighborhood image block are bad pixels, e(N i , N j ) represents the mask matrix of the valid matching pixels of the current image block and the neighborhood image block corresponding to the pixel i, wherein when the matching pixels of the current image block and the neighborhood image block are both non-dead pixels, the value at the corresponding position of the mask matrix is 1, otherwise it is 0.
6. The image processing method according to claim 1, characterized in that: Correcting the bad pixel includes: Calculating weights of various interpolation strategies used by a current region for a bad pixel currently being corrected, wherein the current region is a region with a preset size with the bad pixel currently being corrected as a center pixel, and each interpolation strategy corresponds to an interpolation type; The correction value of the bad pixel currently being corrected is calculated according to the weights of the various interpolation strategies, and correction is performed according to the correction value.
7. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image; A processing module, used to use a same-channel convolution module to perform same-channel convolution processing on the first image to obtain a second image, and use a cross-channel convolution module to perform cross-channel convolution processing on the first image to obtain a third image; perform feature splicing on the second image and the third image to obtain a fourth image; use at least one residual network to process the fourth image to determine the probability value of whether a pixel in the first image is a bad pixel; when the probability value is greater than a preset threshold, detect the pixel as a bad pixel; and correct the bad pixel; wherein the same-channel convolution module includes a first input feature module, a first network module, a second network module, a third network module, a second network module and a first output feature module connected in sequence; the cross-channel convolution module includes a second input feature module, a fourth network module, a second network module, a first network module, a second network module and a second output feature module connected in sequence; the convolution expansion coefficient of the first network module is 2, and the convolution expansion coefficient of the fourth network module is 1.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.
9. An electronic device, comprising a memory and a processor, characterized in that: The processor executes the method according to any one of claims 1 to 6 by calling the computer program stored in the memory.
