Image processing method and electronic device

Abstract

The present application provides an image processing method and electronic device, which are applied to the field of image processing technology. The method includes: starting a camera, performing expansion processing on the original image data output by the camera, and traversing each pixel in the original image data based on the RAW image after the expansion processing. When traversing the first pixel (any pixel in the original image data), determine the directional gradient information and the detection threshold matrix of the first pixel; and determine whether the first pixel is a bad pixel according to the detection threshold matrix of the first pixel; if the first pixel is a bad pixel, adjust the original pixel value of the first pixel to the target pixel value. According to the traversal process of the first pixel, after traversing each pixel in the original image data, obtain the target pixel value of each pixel in the original image data, and output the first image. Thereby, the success rate of bad pixel detection and the accuracy of bad pixel correction can be improved, and a better quality image can be obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to an image processing method and electronic device. Background Art

[0002] With the rapid development of electronic devices, users can use electronic devices to capture images. Due to defects in the manufacturing process of image sensors in electronic devices, images captured by image sensors usually have bad pixels. Bad pixels are pixels with incorrect information (i.e., inaccurate pixel values) on the image due to defects in the array process formed by each light collection point on the image sensor, or errors in the process of converting light signals. Such pixels can be called bad pixels. The accuracy of bad pixels detected by the current bad pixel detection algorithm is not high, and missed detection and false detection may occur, which leads to poor accuracy in removing bad pixels and ultimately poor image quality. Summary of the invention

[0003] The present application provides an image processing method, an electronic device, a chip system, a readable storage medium and a computer program product, which can reduce the probability of missed detection or false detection, improve the success rate of bad pixel detection, and help improve the effect of removing bad pixels, thereby obtaining higher quality images.

[0004] In a first aspect, the present application provides an image processing method, the method comprising: starting a camera; obtaining raw image data output by the camera; performing expansion processing on the raw image data to obtain a RAW image after expansion processing; by performing expansion processing on the raw image data output by the camera to obtain a RAW image after expansion processing, it is possible to ensure that the edge pixels of the original image data have neighborhood information.

[0005] On this basis, based on the RAW image after the expansion processing, each pixel point in the original image data is traversed; the traversal process of the first pixel point (which can be any pixel point in the original image data) is taken as an example to illustrate: when traversing the first pixel point, the directional gradient information of the first pixel point is determined, and the first pixel point is any pixel point in the original image data; according to the directional gradient information of the first pixel point, the detection threshold matrix of the first pixel point is determined; it can provide a judgment standard for subsequent bad pixel judgment and improve the success rate of bad pixel detection. Furthermore, according to the detection threshold matrix of the first pixel point, it is determined whether the first pixel point is a bad pixel; if the first pixel point is a bad pixel, the original pixel value of the first pixel point is adjusted to the target pixel value of the first pixel point.

[0006] Furthermore, after traversing each pixel in the original image data, the target pixel value of each pixel in the original image data is obtained; the original pixel value of the bad pixel in the original image data can be replaced so that the difference between the target pixel value of the replaced pixel and the original pixel values ​​of other pixels in the neighborhood of the pixel is smaller, thereby improving the similarity between the pixel and other pixels in the neighborhood of the pixel, thereby achieving the effect of removing bad pixels.

[0007] Furthermore, according to the target pixel value of each pixel in the original image data, the first image is output, and a first image with better quality after bad pixels are removed can be obtained. Thus, the detection threshold matrix of each pixel can be determined in combination with the directional gradient information of each pixel, and the detection threshold matrix of each pixel is used as a basis for judging whether the pixel is a bad pixel, which can improve the success rate of bad pixel detection. Moreover, in the case of determining that a pixel is a bad pixel, the original pixel value of the pixel can be adjusted to obtain a more accurate target pixel value of the pixel, so that the difference between the target pixel value of the pixel and the original pixel values ​​of other pixels in the neighborhood of the pixel is small, and the similarity between the pixel and other pixels in the neighborhood of the pixel is improved, the effect of removing bad pixels is better, the image information of the output first image is more complete, and the image is clearer and more delicate.

[0008] In addition, compared with the prior art, the image processing method of the embodiment of the present application does not need to rely on the bad pixel table provided by the image sensor manufacturer, nor does it need to record and save the location information of the bad pixels in advance. It can meet the bad pixel detection and correction needs in different scenarios, with a lower probability of missed detection or false detection, a higher success rate of bad pixel detection, a better effect of bad pixel correction, and a higher quality of the first image obtained.

[0009] In some possible implementations, when the first pixel is a bad pixel, the original pixel value of the first pixel is adjusted to a target pixel value, including: determining the target pixel value of the first pixel based on the pixel median of the first pixel and the original pixel value of the first pixel; the pixel median of the first pixel is the median of the original pixel values ​​of all pixels in the detection area corresponding to the first pixel; and adjusting the original pixel value of the first pixel to the target pixel value of the first pixel.

[0010] In some possible implementations, a target pixel value of the first pixel is determined based on a pixel median of the first pixel and an original pixel value of the first pixel, including: determining the target pixel value of the first pixel based on the pixel median of the first pixel, a first weight, the original pixel value of the first pixel, and a second weight; the first weight is used to represent the weight ratio corresponding to the pixel median of the first pixel; the second weight is used to represent the weight ratio corresponding to the original pixel value of the first pixel.

[0011] Therefore, through the above-mentioned implementation method of determining the target pixel value of the first pixel point, a more accurate target pixel value of the first pixel point can be obtained, effectively reducing the difference between the target pixel value of the first pixel point and the original pixel values ​​of other pixels in the neighborhood of the first pixel point, and improving the similarity between the first pixel point and other pixels in the neighborhood of the first pixel point, thereby preparing for outputting a first image with higher quality.

[0012] In some possible implementations, determining whether the first pixel is a bad pixel based on a detection threshold matrix of the first pixel includes: permuting one or more elements in the detection threshold matrix of the first pixel to obtain a first matrix; determining a first number based on the first matrix; the first number is the number of elements in the first matrix that are greater than a second preset value; determining that the first pixel is a bad pixel when the first number is greater than or equal to a first threshold; and determining that the first pixel is not a bad pixel when the first number is less than the first threshold.

[0013] In some possible implementations, when the first pixel is not a bad pixel, the method further includes: using the original pixel value of the first pixel as the target pixel value of the first pixel.

[0014] Based on the above description, the present application can obtain the first matrix by performing a permutation process on the detection threshold matrix of the first pixel, and then obtain the first number based on the first matrix, and further determine whether the first pixel is a bad pixel by comparing the first number with the size of the first threshold set in advance, which can improve the accuracy and success rate of determining whether the first pixel is a bad pixel and reduce the probability of false detection. Moreover, in the case of determining that the first pixel is a bad pixel, by adjusting the original pixel value of the first pixel to the target pixel value of the first pixel, the difference between the target pixel value of the first pixel and the original pixel values ​​of other pixels in the neighborhood of the first pixel can be reduced to achieve the effect of removing bad pixels.

[0015] In some possible implementations, one or more elements in a detection threshold matrix of a first pixel are permuted to obtain a first matrix, including: based on the detection threshold matrix of the first pixel, each element in the detection threshold matrix of the first pixel is permuted in the following first element permutation manner to obtain the first matrix: when the value of the first element is less than a second preset value, the value of the first element is replaced by the second preset value; when the value of the first element is greater than or equal to the second preset value and less than or equal to a third preset value, the value of the first element remains unchanged; when the value of the first element is greater than the third preset value, the value of the first element is replaced by the third preset value; wherein the first element is any element in the detection threshold matrix of the first pixel.

[0016] Therefore, by performing a replacement process on one or more elements in the detection threshold matrix of the first pixel point, it can be ensured that the range of the elements in the first matrix after the replacement process is not too large, thereby preparing for the subsequent calculation of the first weight.

[0017] In some possible implementations, when it is determined that the first pixel is a bad pixel, the method further includes: after obtaining the first matrix, determining a first weight based on the elements in the first matrix and the first quantity. For example, the first weight is obtained by calculating the ratio of the sum of the elements in the first matrix to the first quantity. The calculation formula of the first weight is as follows:

[0018] ;

[0019] in, represents the first weight; A represents the first matrix; represents the sum of all elements in the first matrix; Indicates the first quantity.

[0020] In some possible implementations, after the first weight is obtained, the second weight is obtained by calculating the difference between the first preset value and the first weight.

[0021] Therefore, by determining the first weight and the second weight, it is possible to prepare for calculating the target pixel value of the first pixel when the first pixel is a bad pixel.

[0022] In some possible implementations, each pixel point in the original image data is traversed based on the RAW image after expansion processing, including: determining a first window based on the RAW image after expansion processing; the pixel points in the first window are located in the RAW image after expansion processing; based on the first window, each pixel point in the original image data is traversed; wherein, when traversing the first pixel point, the first window is a detection area centered on the first pixel point.

[0023] Therefore, by determining the first window based on the RAW image after the expansion process, and traversing each pixel point in the original image data based on the first window, the probability of missed detection can be reduced and the success rate and accuracy of bad pixel detection can be improved.

[0024] In some possible implementations, the method further includes: arranging the original pixel values ​​of the pixels in the first window in sequence to obtain a first sequence; and determining the median of the first sequence as the pixel median of the first pixel. Therefore, by obtaining the pixel median of the first pixel, it is possible to prepare for calculating the target pixel value of the first pixel when the first pixel is a bad pixel.

[0025] In some possible implementations, determining the directional gradient information of the first pixel includes: calculating the difference between the original pixel value of each pixel in the first window and the original pixel value of the first pixel, respectively, to obtain the directional gradient information of the first pixel. Therefore, by determining the directional gradient information of the first pixel, it is possible to prepare for the subsequent determination of the detection threshold matrix of the first pixel.

[0026] In some possible implementations, determining the detection threshold matrix of the first pixel point according to the directional gradient information of the first pixel point includes: determining the detection threshold matrix of the first pixel point according to the directional gradient information of the first pixel point, a first parameter, and a second matrix; the first parameter is a coefficient of the directional gradient information of the first pixel point; the second matrix is ​​determined based on the second parameter and the unit matrix. Therefore, by determining the detection threshold matrix of the first pixel point, a data basis can be provided for subsequent determination of whether the first pixel point is a bad pixel, and preparation can also be made for subsequent calculation of the first weight value.

[0027] In some possible implementations, the value of the first parameter is positively correlated with the probability of determining whether the first pixel is a bad pixel, and the value of the second parameter is negatively correlated with the probability of determining whether the first pixel is a bad pixel. Therefore, by setting a larger value of the first parameter and a smaller value of the second parameter, the accuracy of the detection threshold matrix of the first pixel can be improved, thereby improving the probability and success rate of determining whether the first pixel is a bad pixel.

[0028] In some possible implementations, starting the camera includes: starting the camera in response to a first operation of the user; outputting the first image includes: outputting the first image in response to a photo-taking operation of the user.

[0029] Therefore, in response to the user's photo-taking operation, the present application can process the original image data through the above-mentioned image processing method, so as to output a first image after removing bad pixels, and the first image has higher quality and better clarity.

[0030] In some possible implementations, outputting the first image further includes: in response to a user opening a camera application, outputting the first image. In this case, the first image is a preview image. Therefore, the present application can process the original image data in the preview scene through the above-mentioned image processing method to obtain a first image with higher quality, thereby improving the user's preview experience.

[0031] In a second aspect, the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory are coupled, the memory is used to store computer programs, and the processor calls instructions so that the electronic device executes the image processing method in the first aspect and any possible design of the first aspect.

[0032] In a third aspect, the present application provides a chip system, which is applied to an electronic device including a memory, a display screen and a sensor; the chip system includes: one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the interface circuit is used to receive signals from the memory and send signals to the processor, the signals including computer codes or instructions stored in the memory; the processor calls the computer code or instructions so that the electronic device executes the image processing method in the first aspect and any possible design of the first aspect.

[0033] Among them, the chip system may include one chip or multiple chips; when the chip system includes multiple chips, this application does not limit parameters such as the type and quantity of the chips.

[0034] In a fourth aspect, the present application provides a readable storage medium, in which codes or instructions are stored. A processor calls the codes or instructions to enable an electronic device to execute the image processing method in the first aspect and any possible design of the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed, enables the image processing method in the first aspect and any possible design of the first aspect to be executed.

[0036] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of an application scenario of an image processing method provided in one embodiment of the present application;

[0039] Figure 2 A schematic diagram of an image effect after bad pixels are removed in an image processing method provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of original image data with bad pixels and a corresponding pixel set according to an image processing method provided by an embodiment of the present application;

[0041] Figure 4A schematic diagram of another type of original image data with bad pixels in an image processing method provided by an embodiment of the present application;

[0042] Figure 5 A flowchart of an image processing method provided in one embodiment of the present application;

[0043] Figure 6 A schematic diagram of the principle of an extended processing process of an image processing method provided in one embodiment of the present application;

[0044] Figure 7 A schematic diagram of a traversal process of a first window of an image processing method provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of an original pixel value of a first pixel point and a pixel set of a first window of an image processing method provided in one embodiment of the present application;

[0046] Fig. 9 A schematic diagram of a pixel set of directional gradient information of a first pixel point of an image processing method provided by an embodiment of the present application;

[0047] Fig.10 A flowchart of an image processing method provided in one embodiment of the present application;

[0048] Fig.11 A schematic diagram of a pixel set of a detection threshold matrix of a first pixel point of an image processing method provided in one embodiment of the present application;

[0049] Fig.12 A schematic diagram of a pixel set of a first matrix of an image processing method provided by an embodiment of the present application;

[0050] Fig.13 A schematic diagram of a target pixel value of a first pixel point and a pixel set of a first window of an image processing method provided in one embodiment of the present application;

[0051] Fig.14 A schematic diagram of a software architecture of an electronic device provided in one embodiment of the present application;

[0052] Fig.15 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0054] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0055] The image processing method provided in the embodiment of the present application can be applied to image processing scenarios of bad pixel detection and / or bad pixel removal to obtain images with higher quality. The image processing method provided in the embodiment of the present application can process images online in real time or offline, which is not limited in the embodiment of the present application.

[0056] In some embodiments, the image processing method provided in the embodiments of the present application can be applied to scenarios such as security monitoring, traffic management, medical imaging, agricultural monitoring, industrial inspection, scientific research, vehicle driving assistance systems, face recognition, robot vision, video and multimedia systems (such as shooting, recording, video calls and video conferencing).

[0057] For example, in the scenario of security monitoring, the image obtained by the image processing method of the embodiment of the present application does not have bad pixels, the image quality is high, and the clarity is good. The image can accurately identify the details in the monitoring screen. For another example, in the scenario of medical imaging, the image obtained by the image processing method of the embodiment of the present application does not have bad pixels, and the image quality is high, which enables medical staff to obtain accurate diagnosis results and treatment plans based on the image. For another example, in the scenario of industrial inspection, the product image obtained by the image processing method of the embodiment of the present application does not have bad pixels, and the quality of the product image is high. The details of the product can be fully retained, and the detection accuracy during product defect inspection can be improved, and defective products can be better prevented from entering the market. For another example, in the scenario of vehicle driving assistance system, the image obtained by the image processing method of the embodiment of the present application does not have bad pixels, so that the vehicle safety system can perform subsequent analysis and processing based on the image more accurately, thereby ensuring the safety of vehicle operation. For another example, in the scenario of face recognition, the image obtained by the image processing method of the embodiment of the present application does not have bad pixels, so that the face can be recognized more accurately, further improving the accuracy of identity verification. For another example, in the scenario of video and multimedia systems, the image obtained using the image processing method of the embodiment of the present application does not have bad pixels, and can provide a higher quality image data basis for subsequent further analysis and processing.

[0058] The image processing method provided in the embodiment of the present application can be applied to an electronic device with a microcontroller unit (MCU), or a single chip microcomputer. The electronic device can be a mobile phone, a folding screen, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a home appliance, a projector, a digital camera, a scanner, a film scanner, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0059] For example, wearable electronic devices can also be called wearable smart devices, which is a general term for wearable devices that are designed and developed by applying wearable technology to intelligently wear daily wear, such as smart glasses, smart gloves and smart watches.

[0060] It should be noted that the electronic device in the embodiment of the present application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, etc. In addition, in the embodiment of the present application, the electronic device may also be an electronic device in the Internet of Things (IoT) system. The main technical feature of the IoT is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0061] Combine the following Figure 1 , an application scenario of the image processing method of an embodiment of the present application is introduced in detail.

[0062] See also Figure 1 , Figure 1 A schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application is shown.

[0063] When the electronic device 100 is unlocked, the desktop of the electronic device 100 may be displayed as follows: Figure 1 The second interface 11 shown in (1) in FIG. Figure 1 As shown in (1), multiple applications are installed in the electronic device 100. The multiple applications may include a camera 101, a contact book, a phone call, a message, a video, a setting, and an album.

[0064] like Figure 1 As shown in (1) of FIG. 1 , the electronic device 100 responds to the user's triggering operation (such as a single click, multiple clicks, or a long press operation) on the camera 101 in the second interface 11, and displays the following Figure 1 The third interface 12 shown in (2) .

[0065] like Figure 1 As shown in (2), the third interface 12 displayed by the electronic device 100 is an interface of the camera 101. The third interface 12 may include multiple shooting modes of the camera 101. The multiple shooting modes may include aperture mode, night scene mode, photo mode, portrait mode, video mode, and professional mode.

[0066] like Figure 1As shown in (2), the third interface 12 is a preview image interface displayed when the electronic device 100 is in the camera mode. The third interface 12 may include a viewfinder 102, a first control 103, a second control 104, a third control 105, and a fourth control 106. In response to the user's triggering operation on different controls in the third interface 12, the electronic device 100 may perform different operations.

[0067] Optionally, the viewfinder 102 is used to display a preview image so that the user can preview the image. Figure 1 As shown in (2), the preview image is a black image after bad pixels are removed, wherein the portion of the black image including the black area exceeds a first preset ratio.

[0068] It should be noted that the above Figure 1 The preview image shown in (2) is only for illustrative purposes and the embodiments of the present application are not limited thereto.

[0069] It can be seen that there are no bad pixels (for example, Figure 1 There are no white bright spots in the black image after removing the bad pixels as shown in (2).

[0070] It should be noted that the image processed by the image processing method of the embodiment of the present application can be Figure 1 In addition to the black image after bad pixels are removed as shown in (2) in the figure, other images may also be used, and the present embodiment of the application does not limit this. For example, the image processed by the image processing method of the present embodiment of the application may also be as follows Figure 2 The white image after bad pixels are removed as shown in (1) in the figure, wherein the portion of the white image including the white area exceeds a second preset ratio, and the second preset ratio and the first preset ratio may be the same ratio or different ratios. For another example, the image processed by the image processing method of the embodiment of the present application may also be as follows: Figure 2 The image after bad pixels are removed as shown in (2), and the image includes the object to be photographed.

[0071] It should also be noted that Figure 2 The objects to be photographed included in the image after bad pixels are removed as shown in (2) are only for illustrative purposes, and the embodiments of the present application are not limited thereto. In actual shooting, based on the shooting needs of the user, the objects to be photographed included in the image after bad pixels are removed in the embodiments of the present application may also include one or more of the following: people, animals, vehicles, scenery, and buildings, etc. In addition, the objects to be photographed in the embodiments of the present application may be in a stationary state or in a moving state, and the embodiments of the present application are not limited thereto.

[0072] Optionally, the first control 103 is used to trigger the electronic device 100 to start the photographing mode in the camera 101. In response to the user's triggering operation (eg, a single click operation) on the first control 103, the user can take pictures in the photographing mode of the electronic device 100.

[0073] It should be noted that the above method of starting the photo mode in the camera 101 is only an exemplary description, and the embodiments of the present application are not limited thereto. For example, when the current shooting mode of the electronic device 100 is the video recording mode, in response to the user sliding to the right, the shooting mode of the electronic device 100 can be switched from the video recording mode to the photo mode.

[0074] It should also be noted that, in actual applications, after the camera 101 is started, the interface displayed by the electronic device 100 is usually an interface in the photo taking mode (for example, Figure 1 The third interface 12 shown in (2) of FIG. 1 is not limited in this embodiment of the present application. For example, according to actual shooting requirements, after the camera 101 is started, the interface displayed by the electronic device 100 can also be set to the interface in the portrait mode.

[0075] Optionally, the second control 104 is used to display the captured image (for example, Figure 1 The figure image displayed in the second control 104 shown in (2) in the figure or the thumbnail of the recorded video.

[0076] Optionally, the third control 105 is a photo control. In response to the user's triggering operation on the third control 105, the electronic device 100 performs a photo operation, and an image consistent with the content displayed in the current viewfinder 102 can be obtained (for example, Figure 1 The black image after removing the bad pixels is displayed in the second control 104 shown in (3).

[0077] It should be noted that if Figure 1 As shown in (2), the third interface 12 may also display the optical magnification of the camera in the electronic device 100. The optical magnification of the camera refers to the ability of the camera to magnify the observed object (i.e., the object to be photographed) on the imaging sensor or film.

[0078] For example, Figure 1 As shown in (2) in the figure, the optical magnification of the camera includes 0.5 times, 1 times (i.e. Figure 1 1x), 3x and 6x as shown in (2) in the figure.

[0079] Alternatively, if Figure 1 As shown in (2), the third interface 12 may further include a fourth control 106. The fourth control 106 is used to trigger switching between the front camera and the rear camera.

[0080] Alternatively, if Figure 1 As shown in (2) in FIG. 1 , in response to the user's triggering operation on the third control 105, the electronic device 100 displays the following Figure 1 The fourth interface 13 shown in (3) .

[0081] Alternatively, if Figure 1 As shown in (3) in FIG. 1 , in response to the user's triggering operation on the second control 104 in the fourth interface 13, the electronic device 100 may display the following Figure 1 The fifth interface 14 shown in (4) in FIG. 1 is used to enable the user to perform subsequent processing on the black image after removing the bad pixels in the fifth interface 14 .

[0082] Alternatively, if Figure 1 As shown in (4), the fifth interface 14 may include a fifth control 107. The fifth control 107 is used to trigger the electronic device 100 to return to the previous interface (for example, Figure 1 The fourth interface 13 shown in (3)).

[0083] It should be noted that the fifth interface 14 may also include controls such as share, edit, favorite and more. In response to the user's triggering operation on different controls, the electronic device 100 may perform different functions. For example, in response to the user's clicking operation on the share control, the electronic device 100 may display a corresponding interface, allowing the user to upload the captured image (for example, Figure 1 The black image after removing the bad pixels shown in (3) in the figure can be shared to other applications besides the camera application.

[0084] Due to defects in the manufacturing process of image sensors in electronic devices (for example, the array process formed by each light collection point on the image sensor), or errors in the process of image sensors converting light signals into electrical signals, images obtained by electronic devices usually have bad pixels (that is, pixels with inaccurate pixel values ​​due to incorrect information in the image).

[0085] Among them, bad pixels include static defective pixels (SDP) and dynamic defective pixels (DDP). Static defective pixels are pixels with inaccurate pixel values ​​due to technical chip defects of image sensors. Dynamic defective pixels are pixels with inaccurate pixel values ​​due to temperature increases or gain increases of image sensors.

[0086] It should be noted that a bad pixel appears as a pixel with abnormal brightness on the image. For a black image, a bad pixel appears as a bright spot on the black image. Figure 3As shown in (1) in FIG. 1 , image A is a black image, which includes two pixels (or bright spots) with abnormal brightness, the first bad pixel and the second bad pixel. For a white image, the bad pixel appears as a dark spot on the white image. For another example, Figure 4 As shown in (1) in , image B is a white image, which includes two pixels with abnormal brightness (or dark spots) - the third bad pixel and the fourth bad pixel. For another example, Figure 4 As shown in (2), image C is an image of the object to be photographed, and image C includes two pixels with abnormal brightness, the fifth bad pixel and the sixth bad pixel (or in other words, the fifth bad pixel and the sixth bad pixel are both pixels with a large difference in original pixel values ​​from their surrounding pixels).

[0087] It should also be noted that the number of bad pixels in the above-mentioned images A, B and C is only for illustrative purposes and is not limited to this in the embodiments of the present application.

[0088] At present, electronic devices use different detection methods to detect static bad pixels and dynamic bad pixels respectively. Electronic devices detect static bad pixels based on the static bad pixel table provided by image sensor manufacturers. The electronic device compares the coordinates of the current pixel in the image with a certain coordinate in the stored static bad pixel table. If the coordinates of the current pixel in the image are consistent with a certain coordinate in the stored static bad pixel table, the electronic device determines that the current pixel is a bad pixel.

[0089] The electronic device can detect dynamic bad pixels by using a neighborhood comparison method, that is, based on extremely bright pixels (i.e., bright spots) or extremely dark pixels (i.e., dark spots) in a neighborhood, the image is detected for bad pixels. Figure 3 For the image A shown in (1), image A is a black image. At this time, the electronic device uses a neighborhood comparison method, and an extremely bright pixel in a neighborhood is a bad pixel (such as Figure 3 The first bad pixel and the second bad pixel shown in (1) in FIG. Figure 3 As shown in (2), in the pixel set of image A, the original pixel value of the first bad pixel is 545, which is different from the neighborhood of the first bad pixel (i.e. Figure 3 The original pixel values ​​of each pixel in the first neighborhood (shown in (2)) (e.g., 352, 343, etc.) are quite different (or the similarity is small). The original pixel value of the second bad pixel is 600, which is different from the neighborhood of the second bad pixel (i.e. Figure 3 The differences in the original pixel values ​​(e.g., 340, 350, etc.) of each pixel point in the second neighborhood (shown in (2)) are relatively large.

[0090] It should be noted that the neighborhood refers to a set of pixels within a certain range around a specific pixel in an image, for example, Figure 3 The size of the first neighborhood shown in (2) is 3*3, and the shape of the first neighborhood is a square. In addition, the size and shape of the neighborhood depend on the requirements of the algorithm and the purpose of image processing. Figure 3 The size and shape of the first neighborhood and the size and shape of the second neighborhood shown in (2) are only exemplary and are not limited here.

[0091] In the prior art, electronic devices can use the same correction method to correct static bad pixels and dynamic bad pixels respectively. After detecting a bad pixel (i.e., a static bad pixel or a dynamic bad pixel), the electronic device can use a method such as median filtering to correct the bad pixel (e.g., the current pixel mentioned above). In other words, the electronic device can directly replace the original pixel value (or bad pixel value) of the bad pixel with the median value of other pixels in the neighborhood of the bad pixel to achieve the purpose of bad pixel correction.

[0092] However, for static bad pixels, electronic devices must rely on the static bad pixel table provided by the image sensor manufacturer, otherwise they cannot detect and correct static bad pixels, which is very limited. In addition, since the memory of the electronic device to store the static bad pixel table is limited, it may not be able to completely cover all static bad pixels, and it is impossible to correct static bad pixels indefinitely. For dynamic bad pixels, although the number of dynamic bad pixels that can be corrected by electronic devices is not limited, it is easy to miss detection, misdetection, and loss of edge information, resulting in low accuracy of dynamic bad pixel detection.

[0093] In addition, after detecting a bad pixel (i.e., a static bad pixel or a dynamic bad pixel), the electronic device directly uses the median value of other pixels in the bad pixel's neighborhood to replace the original pixel value of the bad pixel. However, the actual pixel value of the bad pixel obtained by this correction method (i.e., the median of the original pixel values ​​of other pixels in the bad pixel's neighborhood) is not accurate enough. In other words, the actual pixel value of the bad pixel may still be significantly different from the original pixel values ​​of other pixels in its neighborhood, resulting in poor bad pixel correction effect, low image clarity, and poor image quality (for example, Figure 3 The first bad pixel and the second bad pixel in the image A shown in (1) still exist; for example, Figure 4 The third and fourth bad pixels in the image B shown in (1) still exist; for example, Figure 4 The fifth and sixth bad pixels in image C shown in (2) still exist).

[0094] In view of this, the image processing method provided in the embodiment of the present application can ensure that the edge pixels of the original image data have neighborhood information by performing expansion processing on the original image data output by the camera to obtain the RAW image after the expansion processing. On this basis, each pixel in the original image data is traversed based on the RAW image after the expansion processing. Take the traversal process of the first pixel (which can be any pixel in the original image data) as an example to illustrate: when traversing the first pixel, determine the directional gradient information of the first pixel, and determine the detection threshold matrix of the first pixel based on the directional gradient information of the first pixel, which can provide a judgment standard for subsequent bad pixel judgment and improve the success rate of bad pixel detection. Furthermore, according to the detection threshold matrix of the first pixel, determine whether the first pixel is a bad pixel; if the first pixel is a bad pixel, adjust the original pixel value of the first pixel to the target pixel value of the first pixel. Furthermore, after traversing each pixel in the original image data, the target pixel value of each pixel in the original image data is obtained, and the original pixel value of the bad pixel in the original image data can be replaced, so that the difference between the target pixel value of the replaced pixel and the original pixel values ​​of other pixels in the neighborhood of the pixel is small, and the similarity between the pixel and other pixels in the neighborhood of the pixel is improved, so as to achieve the effect of removing the bad pixel. Furthermore, according to the target pixel value of each pixel in the original image data, the first image is output, and a first image with better quality after removing the bad pixel can be obtained.

[0095] Combination of the above Figure 1 and Figure 2 This article introduces an application scenario of an embodiment of the present application and briefly introduces the implementation process of image processing in current electronic devices. Figures 5 to 13 , describes in detail the specific implementation process of an image processing method provided in an embodiment of the present application.

[0096] See also Figure 5 , Figure 5 A flow chart of an image processing method provided in an embodiment of the present application is shown.

[0097] like Figure 5 As shown, the image processing method of the embodiment of the present application specifically includes the following steps:

[0098] S101, start the camera.

[0099] Optionally, in some embodiments, the camera may be a front camera or a rear camera of the electronic device, the front camera may include one or more cameras, and the rear camera may also include one or more cameras.

[0100] It should be noted that, in actual applications, the electronic device may determine whether to start the front camera or the rear camera according to the actual usage scenario, and the embodiments of the present application do not limit this. Figure 1 As shown in (2), in the scene of the photo mode, the electronic device starts the camera by default and is set to start the rear camera of the electronic device. At this time, in response to the user clicking the fourth control 106, the electronic device can switch from the rear camera to the front camera.

[0101] Optionally, in some embodiments, the electronic device may activate the camera in response to a first operation of the user. For example, the first operation may be an operation of the user opening a camera application. In the case where the first operation is an operation of the user opening a camera application, the electronic device may activate the camera in response to the operation of the user opening the camera application (e.g., Figure 1 In the second interface 11 shown in (1), the user triggers the camera 101 to start the camera.

[0102] It should be noted that the above-mentioned first operation is only used as an exemplary description, and the embodiments of the present application are not limited thereto. Those skilled in the art can start the camera through other first operations. For example, the first operation can also be a voice command of "start camera" input by the user. Upon receiving the voice command of "start camera" input by the user, the electronic device can start the camera. For another example, the first operation is an operation in which the user calls the camera function in a third-party application. At this time, the electronic device can start the camera in response to the user's operation of calling the camera function in the third-party application.

[0103] Optionally, in some embodiments, when the camera is started, the electronic device can display a first interface. By displaying the first interface, the electronic device can prepare for subsequent interactive operations of the user.

[0104] Optionally, in some embodiments, the first interface is a photo preview interface. Figure 1 The third interface 12 shown in (2) .

[0105] It should be noted that, in actual applications, electronic devices usually set the interface displayed when the camera is started to be the interface in the photo mode by default. Those skilled in the art may also set the interface displayed when the camera is started to be the interface in other shooting modes (e.g., portrait mode) other than the photo mode according to actual use requirements, and the embodiments of the present application do not limit this.

[0106] In summary, by starting the camera, the electronic device can prepare for the subsequent acquisition of raw image data (RAW image format, RAW), thereby providing a data basis for the image processing method of the embodiment of the present application.

[0107] S102, obtaining original image data output by the camera.

[0108] Optionally, in some embodiments, the original image data, or the original RAW image, is the unprocessed data captured by the image sensor in the electronic device. The original image data contains rich image detail information and dynamic range information, and can provide a good data basis for subsequent image processing.

[0109] After starting the camera, the electronic device can obtain the original image data output by the camera, thereby providing a data basis for subsequent image processing (including bad pixel detection and bad pixel correction).

[0110] S103, performing expansion processing on the original image data to obtain a RAW image after expansion processing.

[0111] Considering that there are multiple edge pixels in the original image data, none of the multiple edge pixels have complete neighborhood information, when filtering or other processing is performed on the original image data, blur or discontinuity may appear at the edge of the original image data. Therefore, after obtaining the original image data, the electronic device may perform expansion processing on the original image data to ensure that each edge pixel in the original image data has complete neighborhood information, thereby avoiding the loss of edge details of the original image data when the electronic device performs subsequent processing such as filtering on the original image data.

[0112] After obtaining the original image data, the electronic device can perform expansion processing on the original image data to obtain an expanded RAW image. By obtaining an expanded RAW image, the embodiment of the present application can ensure that multiple edge pixels in the original image data have complete neighborhood information, thereby preparing for subsequent image processing.

[0113] S104, traversing each pixel point in the original image data based on the RAW image after the expansion processing.

[0114] After obtaining the RAW image after the expansion process, the electronic device can traverse each pixel point in the original image data based on the RAW image after the expansion process.

[0115] Optionally, in some embodiments, the electronic device can determine the first window based on the RAW image after the expansion process, and traverse each pixel in the original image data based on the first window. By determining the first window, the embodiment of the present application can provide a data basis for the neighborhood pixels for subsequent image processing, thereby improving the accuracy of bad pixel detection and bad pixel correction.

[0116] Optionally, the pixels in the first window are located in the RAW image after the expansion process. The first window is used to slide on the RAW image after the expansion process and perform local operations. The size of the first window includes the width and height of the first window. Exemplarily, the size of the first window is 3*3, that is, the width of the first window is 3 pixels and the height is also 3 pixels.

[0117] In summary, the electronic device can determine whether each pixel in the original image data is a bad pixel and determine the target pixel value of each pixel in the original image data when traversing each pixel in the original image data. Furthermore, by determining the target pixel value of each pixel in the original image data, the bad pixels in the original image data can be corrected to achieve the effect of removing the bad pixels in the original image data, and can output a first image with higher clarity and higher image quality (i.e., the image after removing the bad pixels, such as Figure 1 The black image after removing bad pixels as shown in (2) in the figure).

[0118] Considering that the electronic device performs the same process of traversing each pixel point in the original image data, the following describes the traversal process of the first pixel point in the original image data as an example in combination with S1041 to S1045:

[0119] S1041, when traversing the first pixel point, determining directional gradient information of the first pixel point.

[0120] Optionally, when traversing the first pixel point, the first window is a detection area centered on the first pixel point. The first pixel point is any pixel point in the original image data. In other words, the first pixel point can be an edge pixel point in the original image data, or a non-edge pixel point in the original image data.

[0121] Optionally, in some embodiments, the directional gradient information of the first pixel refers to the gradient information of the first pixel in various directions. For example, the directional gradient information of the first pixel includes one or more of the following: the gradient in the horizontal direction of the first pixel, the gradient in the vertical direction of the first pixel, and the gradient in the diagonal direction of the first pixel (for example, the 45° direction and the 135° direction of the first pixel), etc.

[0122] Optionally, in the process of traversing the first pixel point in the original image data based on the RAW image after expansion processing, the electronic device can obtain the directional gradient information of the first pixel point by calculating the difference between the original pixel value of the first pixel point and the original pixel values ​​of the pixels in each direction of the first pixel point.

[0123] It should be noted that if the difference between the original pixel value of the first pixel and the original pixel values ​​of the pixels in all directions of the first pixel is large, it means that the gradients of the first pixel and the pixels in all directions of the first pixel are large and the similarities are small. At this time, the possibility that the first pixel is a bad pixel is high.

[0124] In summary, the electronic device can obtain the directional gradient information of the first pixel point, and by determining the directional gradient information of the first pixel point, it can provide a data basis for subsequently determining the detection threshold matrix of the first pixel point.

[0125] S1042: Determine a detection threshold matrix of the first pixel point according to the directional gradient information of the first pixel point.

[0126] Optionally, in some embodiments, the detection threshold matrix of the first pixel is a basis for determining whether the first pixel is a bad pixel.

[0127] Optionally, in some embodiments, after determining the directional gradient information of the first pixel, the electronic device may determine a detection threshold matrix of the first pixel based on the directional gradient information of the first pixel, the first parameter and the second matrix.

[0128] Optionally, in some embodiments, the first parameter is a coefficient of the directional gradient information of the first pixel. The first parameter is used to represent the growth rate of the directional gradient information of the first pixel. Exemplarily, the value range of the first parameter can be (0, 0.1). The specific value of the first parameter can be 0.02.

[0129] Optionally, in some embodiments, the second matrix is ​​determined based on the second parameter and the unit matrix. The second parameter is a coefficient of the unit matrix. The second parameter is used to represent the growth rate of the unit matrix. Exemplarily, the value range of the second parameter can be [-10, 0). The specific value of the second parameter can be -3.1.

[0130] It should be noted that the above-mentioned value range and specific value of the first parameter, and the value range and specific value of the second parameter are exemplary descriptions, and the embodiments of the present application do not limit this. That is to say, in the actual image processing process, those skilled in the art can set different value ranges of the first parameter and the value ranges of the second parameter, and adjust the value of the first parameter and / or the value of the second parameter to achieve different degrees of bad pixel removal effects.

[0131] Optionally, in some embodiments, the expression of the detection threshold matrix of the first pixel is as follows:

[0132] ;

[0133] ;

[0134] in, A detection threshold matrix representing the first pixel; Represents the directional gradient information of the first pixel; represents the absolute value of the directional gradient information of the first pixel; slope represents the first parameter; M represents the second matrix; offset represents the second parameter, and E represents the unit matrix.

[0135] For example, when the slope is 0.02 and the offset is -3.1, , hour, .

[0136] Optionally, in some embodiments, when the value of the second parameter remains unchanged, the value of the first parameter is positively correlated with the probability of determining whether the first pixel is a bad pixel. That is, when the value of the second parameter remains unchanged, the larger the value of the first parameter, the greater the probability that the electronic device determines whether the first pixel is a bad pixel, that is, the more accurate the detection threshold matrix of the first pixel is.

[0137] Optionally, in some embodiments, when the value of the first parameter remains unchanged, the value of the second parameter is negatively correlated with the probability of determining whether the first pixel is a bad pixel. That is, when the value of the first parameter remains unchanged, the smaller the value of the second parameter, the greater the probability that the electronic device determines whether the first pixel is a bad pixel, that is, the more accurate the detection threshold matrix of the first pixel is.

[0138] It can be seen that when the value of the first parameter is larger and the value of the second parameter is smaller, the probability that the electronic device determines whether the first pixel is a bad pixel is greater, that is, the detection threshold matrix of the first pixel is more accurate.

[0139] In summary, the electronic device can obtain the detection threshold matrix of the first pixel. The embodiment of the present application can prepare for the subsequent determination of whether the first pixel is a bad pixel by determining the detection threshold matrix of the first pixel.

[0140] S1043, determining whether the first pixel is a bad pixel according to the detection threshold matrix of the first pixel.

[0141] After obtaining the detection threshold matrix of the first pixel, the electronic device can determine whether the first pixel is a bad pixel according to the detection threshold matrix of the first pixel. If the first pixel is a bad pixel, the electronic device executes S1044. If the first pixel is not a bad pixel, the electronic device executes S1045.

[0142] Optionally, in some embodiments, the electronic device may perform a permutation process (or a linear transformation process) on one or more elements in the detection threshold matrix of the first pixel point to obtain a first matrix, and determining the first matrix can provide a data basis for subsequent processing.

[0143] In other words, the first matrix is ​​a matrix obtained by performing a permutation process on the detection threshold matrix of the first pixel. For example, the first matrix can be .

[0144] Furthermore, the electronic device may determine the first number based on the first matrix, and by determining the first number, it may prepare for the subsequent determination of whether the first pixel is a bad pixel.

[0145] Optionally, the first number is the number of elements in the first matrix that are greater than a second preset value. The second preset value is a value pre-set in the electronic device.

[0146] Exemplarily, the second preset value may be 0. When the second preset value is 0 and the first matrix is , the first quantity is 7.

[0147] It should be noted that the specific value of the first number is positively correlated with the size of the first window (i.e., the width and height of the first window) mentioned above. In other words, the larger the size of the first window, the larger the value of the first number. For example, when the width and height of the first window are both 3 pixels, the value of the first number is 7. For another example, when the width and height of the first window are both 5 pixels, the value of the first number is 21. For another example, when the width and height of the first window are both 7 pixels, the value of the first number is 45.

[0148] Furthermore, the electronic device may determine whether the first pixel is a bad pixel by comparing the first number with the first threshold. When the first number is greater than or equal to the first threshold, the first pixel is determined to be a bad pixel. That is, the difference between the original pixel value of the first pixel and the original pixel value of the pixel in the neighborhood of the first pixel is large, and at this time, the electronic device executes S1044. When the first number is less than the first threshold, it is determined that the first pixel is not a bad pixel. That is, the difference between the original pixel value of the first pixel and the original pixel value of the pixel in the neighborhood of the first pixel is small, and at this time, the electronic device executes S1045.

[0149] Optionally, the first threshold is a bad pixel determination threshold pre-set in the electronic device. Exemplarily, the first threshold may be 7.

[0150] It should be noted that the above second preset value and the first threshold value are exemplary descriptions, and the present application embodiment does not limit this. In practical applications, those skilled in the art can adjust the second preset value and / or the first threshold value according to image processing requirements.

[0151] In summary, the electronic device can determine whether the first pixel is a bad pixel, and perform subsequent corresponding processing according to the determination result.

[0152] S1044: When the first pixel is a bad pixel, adjust the original pixel value of the first pixel to the target pixel value of the first pixel.

[0153] Optionally, in some embodiments, when the first pixel is determined to be a bad pixel, the difference between the original pixel value of the first pixel and the original pixel values ​​of the pixels in the neighborhood of the first pixel is large. Therefore, the electronic device can determine the target pixel value of the first pixel based on the pixel median of the first pixel and the original pixel value of the first pixel, and adjust the original pixel value of the first pixel to the target pixel value of the first pixel.

[0154] Optionally, in some embodiments, the pixel median of the first pixel is the median of the original pixel values ​​of all pixels in the detection area corresponding to the first pixel (or the median filtering result). ) can be 377.8.

[0155] It should be noted that the detection area corresponding to the first pixel point is equivalent to the first window described in S104.

[0156] Optionally, in some embodiments, based on the description of the first window in S104, after determining the first window, the electronic device may also sequentially arrange the original pixel values ​​of the pixels in the first window to obtain a first sequence, and determine the median of the first sequence as the pixel median of the first pixel. By determining the pixel median of the first pixel, the embodiment of the present application can provide a data basis for the subsequent calculation of the target pixel value of the first pixel when determining that the first pixel is a bad pixel.

[0157] It should be noted that the electronic device may arrange the original pixel values ​​of the pixels in the first window in order from large to small or from small to large, and the embodiment of the present application is not limited to this.

[0158] Optionally, in another embodiment, the electronic device may also determine the target pixel value of the first pixel based on the pixel mean value of the first pixel and the original pixel value of the first pixel, and adjust the original pixel value of the first pixel to the target pixel value of the first pixel. The pixel mean value of the first pixel is the average value of the original pixel values ​​of all pixels in the detection area corresponding to the first pixel.

[0159] In summary, the electronic device can obtain the target pixel value of the first pixel. The embodiment of the present application adjusts the original pixel value of the first pixel to the target pixel value of the first pixel, so that the difference between the actual pixel value of the first pixel, that is, the target pixel value of the first pixel and the original pixel value of the pixel in the neighborhood of the first pixel is smaller, thereby achieving the purpose of bad pixel correction and improving the quality of the output image.

[0160] S1045: When the first pixel is not a bad pixel, use the original pixel value of the first pixel as the target pixel value of the first pixel.

[0161] When the first pixel is not a bad pixel, the difference between the original pixel value of the first pixel and the original pixel values ​​of the pixels in the neighborhood of the first pixel is small. Therefore, the electronic device can directly output the original pixel value of the first pixel as the target pixel value of the first pixel without adjustment.

[0162] S105, after traversing each pixel point in the original image data, obtaining a target pixel value of each pixel point in the original image data.

[0163] The electronic device may traverse each pixel in the original image data according to the execution process of S1041 to S1045. After traversing each pixel in the original image data, the electronic device may obtain the target pixel value of each pixel in the original image data. The embodiment of the present application can provide a data basis for outputting a high-quality image by obtaining the target pixel value of each pixel in the original image data.

[0164] S106: output a first image according to the target pixel value of each pixel in the original image data.

[0165] After obtaining the target pixel value of each pixel in the original image data, the electronic device may output the first image according to the target pixel value of each pixel in the original image data.

[0166] Optionally, in some embodiments, the first image may be a preview image. In response to a user opening a camera application (e.g., Figure 1 The electronic device may display a first image (e.g., Figure 1 The black image after removing the bad pixels is displayed in the viewfinder 102 shown in (2).

[0167] Optionally, in another embodiment, the first image may be a photographic image. Figure 1 In response to the user triggering operation on the third control 105 as shown in (2) in FIG. 1 ), the electronic device may output a first image (for example, Figure 1 The black image after removing the bad pixels is displayed in the second control 104 shown in (3).

[0168] It should be noted that the above first image is only an exemplary description, and the embodiments of the present application are not limited thereto. Figure 2 The white image after removing bad pixels as shown in (1) in the figure, or Figure 2 The image after bad pixels are removed as shown in (2) in FIG. 1 . For another example, the first image may also be as follows: Figure 1 Any frame image in the video obtained in the recording mode shown in (2).

[0169] It should also be noted that, in the case where the first image is any frame of image in a video, the electronic device can perform S101 to S106 on any frame of original image data during the process of recording the video to obtain the first image after bad pixels are removed.

[0170] In summary, by executing S101 to S106, the electronic device can detect and remove bad pixels in the original image data, and the effect of bad pixel removal (or bad pixel correction) is better, and the obtained first image has better clarity and higher quality.

[0171] The image processing method of the embodiment of the present application can obtain the original image data output by the camera by starting the camera, and provide a data basis for subsequent image processing (including bad pixel detection and bad pixel correction). Then, the original image data is extended to obtain a RAW image after extension processing. By extending the original image data, it can be ensured that the edge pixel points of the original image data have neighborhood information, and thus preparations can be made for subsequent image processing.

[0172] Furthermore, based on the RAW image after the expansion processing, each pixel in the original image data is traversed. When traversing the first pixel (which can be any pixel in the original image data), the directional gradient information of the first pixel is determined; and based on the directional gradient information of the first pixel, the detection threshold matrix of the first pixel is determined, which can provide a judgment basis for subsequent bad pixel detection.

[0173] Furthermore, based on the detection threshold matrix of the first pixel point, it is possible to determine whether the first pixel point is a bad pixel; when the first pixel point is a bad pixel point, the original pixel value of the first pixel point is adjusted to the target pixel value of the first pixel point, and by adjusting the original pixel value of the first pixel point, the actual pixel value of the first pixel point, that is, the difference between the target pixel value of the first pixel point and the original pixel values ​​of other pixels in the neighborhood of the first pixel point, is reduced, thereby improving the similarity between the first pixel point and other pixels in the neighborhood of the first pixel point.

[0174] Based on the above description, after traversing each pixel in the original image data, the target pixel value of each pixel in the original image data can be obtained, and each pixel in the original image data has a high similarity with other pixels in the neighborhood of the corresponding pixel. Finally, according to the target pixel value of each pixel in the original image data, the first image (for example, Figure 1 The black image after removing bad pixels as shown in (2) in the figure can obtain an image with higher definition and better quality.

[0175] Therefore, by combining the directional gradient information of each pixel point, the detection threshold matrix of each pixel point is determined, and the detection threshold matrix of each pixel point is used as the basis for judging whether the pixel point is a bad pixel, which can improve the success rate of bad pixel detection. In addition, when a pixel point is determined to be a bad pixel point, the original pixel value of the pixel point can be adjusted to obtain a more accurate target pixel value of the pixel point, so that the difference between the target pixel value of the pixel point and the original pixel values ​​of other pixels in the neighborhood of the pixel point is smaller, and the similarity between the pixel point and other pixels in the neighborhood of the pixel point is improved, so as to achieve a better effect of removing bad pixels, and the image information of the output first image is more complete, and the image is clearer and more delicate.

[0176] Compared with the prior art, the image processing method of the embodiment of the present application traverses each pixel in the original image data and determines whether the pixel is a bad pixel through the detection threshold matrix corresponding to each pixel, so as to process multiple continuous bad pixels (including static bad pixels and dynamic bad pixels) in the original image data. Moreover, when processing static bad pixels, the image processing method of the embodiment of the present application does not need to rely on the static bad pixel table provided by the image sensor manufacturer, nor does it need to record and save the location information of the static bad pixels in advance, so as to realize the detection and correction of static bad pixels. In addition, when processing dynamic bad pixels (for example, bad pixels that appear during the use of the image sensor), the image processing method of the embodiment of the present application traverses each pixel in the original image data and uses the detection threshold matrix of each pixel as the basis for judging whether the pixel is a bad pixel, so as to reduce the probability of missed detection or false detection, meet the bad pixel detection and correction requirements in different scenarios, have a higher success rate of bad pixel detection, and have a better effect of bad pixel correction. The obtained first image has a higher quality than the image obtained by the current image processing method. For example, Figure 1 The black image after removing the bad pixels shown in (4) is compared with Figure 3 For the image A shown in (1), the first and second bad pixels are removed, resulting in higher definition and better quality. For another example, Figure 2 The white image after removing bad pixels shown in (1) is compared with Figure 4 For the image B shown in (1), the third and fourth bad pixels are removed, resulting in higher definition and better quality. For another example, Figure 2 The image of the person shown in (2) after bad pixels are removed is compared to Figure 4 As for the image C shown in (2), the fifth and sixth bad pixels are removed, and the clarity and quality are higher.

[0177] Based on the relevant description of S103, the electronic device may adopt multiple possible implementations to perform expansion processing on the original image data. A possible implementation of the electronic device performing expansion processing on the original image data is described in detail below.

[0178] Optionally, in some embodiments, after obtaining the original image data, the electronic device can perform expansion processing on the original image data based on the first preset size and in a preset expansion direction to obtain a RAW image after expansion processing. The embodiment of the present application can ensure that multiple edge pixels in the original image data have complete neighborhood information by performing expansion processing on the original image data to obtain a RAW image after expansion processing, thereby reducing misjudgment, improving the accuracy of bad pixel detection, and improving the effect of bad pixel removal of the original image data, and finally outputting a higher quality image.

[0179] Optionally, in some embodiments, the first preset size is a pixel width for expanding the original image data stored in advance in the electronic device. The first preset size may include one pixel or multiple pixels. Exemplarily, the first preset size may be 2 pixels or 3 pixels.

[0180] Optionally, in some embodiments, the preset expansion direction is a direction sequence for expanding the original image data stored in advance in the electronic device. Exemplarily, the preset expansion direction may be the clockwise direction of the original image data; or, the preset expansion direction may also be the counterclockwise direction of the original image data; or, the preset expansion direction may also be other expansion directions pre-set in the electronic device except the clockwise direction of the original image data and the counterclockwise direction of the original image data.

[0181] Optionally, in some embodiments, when expanding the original image data, the electronic device needs to set the pixel length for expanding the original image data in addition to the first preset size in advance, so as to ensure that each edge pixel point in the original image data has complete neighborhood information. Exemplarily, the pixel length for expanding the original image data set in advance in the electronic device can be the same as the pixel length of the original image data.

[0182] In order to facilitate those skilled in the art to understand the above-mentioned extended processing method, Figure 6 , which details the specific implementation process of electronic equipment performing expansion processing on original image data.

[0183] like Figure 6 As shown in (1) in , the size of the original image data is N*N, that is, the width of the original image data is N pixels, and the length is also N pixels, where N is an integer greater than 1. Figure 6 As shown in (1) in FIG. 1 , the preset expansion direction is the clockwise direction of the original image data formed by the dotted arrow. Figure 6 As shown in (2) in , the first preset size is R pixels, where R is an integer greater than or equal to 1 and less than or equal to N. Figure 6 As shown in (2), the pixel length of the expanded original image data is N pixels.

[0184] like Figure 6 As shown in (2) in FIG. 1 , the width of the first area to be expanded is a first preset size, that is, R pixels, and the length is N pixels. The electronic device can copy the original pixel values ​​corresponding to all pixels in the first area to be expanded in the original image data. After completing the copying, the electronic device can move the first area to be expanded along the first preset size. Figure 6The original image data is expanded for the first time in the direction of the arrow shown in (2) in FIG. The electronic device expands the original image data for the first time in the first region as shown in FIG. Figure 6 As shown in (3) in FIG. Based on this, the electronic device can perform a second expansion on the original image data.

[0185] Furthermore, if Figure 6 As shown in (3) in FIG. 1 , the width of the second area to be expanded is a first preset size, that is, R pixels, and the length is N+R pixels. The electronic device can copy the original pixel values ​​corresponding to all pixels in the second area to be expanded in the original image data. After completing the copying, the electronic device can move the second area to be expanded along the path as shown in FIG. Figure 6 The original image data is expanded for the second time in the direction of the arrow shown in (3) in FIG. The second area after the electronic device expands the original image data for the second time is as shown in FIG. Figure 6 As shown in (4) in FIG. Based on this, the electronic device can perform a third expansion on the original image data.

[0186] Furthermore, if Figure 6 As shown in (4) in FIG. 1 , the width of the third area to be expanded is a first preset size, that is, R pixels, and the length is N+R pixels. The electronic device can copy the original pixel values ​​corresponding to all pixels in the third area to be expanded in the original image data. After completing the copying, the electronic device can move the third area to be expanded along the path as shown in FIG. Figure 6 The original image data is expanded for the third time in the direction of the arrow shown in (4) in FIG. The third area after the electronic device expands the original image data for the third time is as shown in FIG. Figure 6 As shown in (5) in FIG. Based on this, the electronic device can perform a fourth expansion on the original image data.

[0187] Furthermore, if Figure 6 As shown in (5) in FIG. 1 , the width of the fourth area to be expanded is a first preset size, that is, R pixels, and the length is N+2R pixels. The electronic device can copy the original pixel values ​​corresponding to all pixels in the fourth area to be expanded in the original image data. After completing the copying, the electronic device can move the fourth area to be expanded along the path as shown in FIG. Figure 6 The original image data is expanded for the fourth time in the direction of the arrow shown in (5) in FIG. The fourth area after the electronic device expands the original image data for the fourth time is as shown in FIG. Figure 6 As shown in (6) in .

[0188] In summary, after completing the four expansions, the electronic device can be obtained as follows Figure 6The RAW image after the expansion processing shown in (6) has a width of N+2R pixels and a length of N+2R pixels. By performing expansion processing on the original image data, the embodiment of the present application can ensure that each edge pixel in the original image data has complete neighborhood information, thereby preparing for subsequent image processing.

[0189] It should be noted that, in practical applications, those skilled in the art can select the corresponding first preset size based on the actual size of the original image data, and the embodiment of the present application does not specifically limit the actual size of the original image data and the first preset size. For example, when the actual size of the original image data is 1026*1026, the first preset size is 2 pixels.

[0190] Based on the above description, after obtaining the RAW image after the expansion process, the electronic device may execute S104.

[0191] Based on the relevant description of S104 and S1041, Figure 6 The obtained RAW image after the expansion processing introduces in detail a possible implementation method of the electronic device determining the directional gradient information of each pixel point in the original image data.

[0192] Based on the relevant description of S104, the electronic device can traverse each pixel point in the original image data based on the first window in a preset traversal direction, that is, the pixel point where the electronic device starts to traverse is the first pixel point in the upper left corner of the original image data.

[0193] For example, Figure 7 As shown, the size of the RAW image after the expansion process is (N+2R)*(N+2R). The first window is the first 3*3 solid line small square. The pixel point where the electronic device starts to traverse is Figure 7 The first pixel point shown is located at the center of the first solid line small square, that is, Figure 7 The preset traversal direction is as follows Figure 7 The dashed arrow direction is from left to right as shown.

[0194] It should be noted that the smaller the size of the first window is, the higher the accuracy of sliding on the RAW image after the expansion process and performing local operations is, and the faster the operation speed is.

[0195] It should also be noted that the size of the first window can be the same as the first preset size, for example, the width and height of the size of the first window are both 3 pixels, and the first preset size is also 3 pixels. Alternatively, the size of the first window can be different from the first preset size, for example, the width and height of the size of the first window are both 3 pixels, and the first preset size is 2 pixels, which is not limited in the embodiments of the present application.

[0196] It should be understood that the words "left", "right" and "up" above indicating directions are based on Figure 7 Directions shown are described.

[0197] Considering that when traversing the first pixel point in the original image data, the first window is a detection area centered on the first pixel point, therefore, the width and height of the first window are usually the same. In addition, the first window can also be called a template. When the width and height of the first window are the same, the size of the first window, that is, the width and height of the first window, can be collectively referred to as a stencil radius (Sr).

[0198] It should be noted that the above Figure 7 The shape and size of the first window shown are only exemplary. Usually, in order to ensure that the first pixel is at the center of the first window, the shape of the first window is usually a square, that is, the width and height of the first window are the same. In practical applications, those skilled in the art can set the first window of other shapes (for example, a circle) and sizes (for example, a template radius of 5 pixels) according to the calculation requirements, and the embodiments of the present application are not limited to this.

[0199] Since the specific implementation process of the electronic device determining the directional gradient information of each pixel in the original image data is the same, the following is a detailed description using an example of the electronic device determining the directional gradient information of the first pixel in the original image data.

[0200] Optionally, when traversing the first pixel point based on the first window, the electronic device may respectively calculate the difference between the original pixel value of each pixel point in the first window and the original pixel value of the first pixel point to obtain the directional gradient information of the first pixel point.

[0201] For example, Figure 8 As shown, the original pixel value of the first pixel is 545, and the pixel set of the first window is as follows Figure 8 When the pixel set shown is used, the pixel set of the directional gradient information of the first pixel point calculated by the electronic device is as follows: Fig. 9 shown.

[0202] It should be noted that Figure 7The first pixel point is taken as an edge pixel point in the original image data for illustration, but the embodiments of the present application are not limited to this, that is, the first pixel point may also be a non-edge pixel point in the original image data.

[0203] In summary, the electronic device can obtain the directional gradient information of the first pixel point, and by determining the directional gradient information of the first pixel point, it can provide a data basis for subsequently determining the detection threshold matrix of the first pixel point.

[0204] Based on the relevant description of S1044, the following Figures 10 to 13 , describes in detail a possible implementation method in which an electronic device determines a target pixel value of a first pixel when the first pixel is a bad pixel.

[0205] See also Fig.10 , Fig.10 A flow chart of an image processing method provided in an embodiment of the present application is shown.

[0206] like Fig.10 As shown, the image processing method of the embodiment of the present application specifically includes the following steps:

[0207] S201: After obtaining a first matrix, determine a first weight based on elements in the first matrix and a first quantity.

[0208] Optionally, the first weight is used to represent the weight ratio corresponding to the pixel median of the first pixel. Exemplarily, the value range of the first weight (α) is (0, 1). The specific value of the first weight may be 0.76.

[0209] Based on the relevant description of S1043, after performing a permutation process on one or more elements in the detection threshold matrix of the first pixel point, the electronic device can obtain a first matrix. For example, the pixel set of the directional gradient information of the first pixel point is as follows Fig. 9 When the pixel set shown in FIG. 1 is used, the pixel set of the detection threshold matrix of the first pixel point is obtained as follows: Fig.11 Further based on Fig.11 The pixel set of the detection threshold matrix of the first pixel point shown in FIG. Fig.12 shown.

[0210] Optionally, in some embodiments, the electronic device may, based on the detection threshold matrix of the first pixel point, perform a permutation process on each element in the detection threshold matrix of the first pixel point according to the following first element (value) permutation method to obtain a first matrix:

[0211] The electronic device may first compare the value of the first element in the detection threshold matrix of the first pixel with the second preset value pre-stored in the electronic device to determine whether the value of the first element needs to be replaced. If the value of the first element is less than the second preset value, the electronic device may replace the value of the first element with the second preset value.

[0212] When the value of the first element is greater than the second preset value, the electronic device may further compare the value of the first element with a third preset value pre-stored in the electronic device to determine again whether the value of the first element needs to be replaced.

[0213] Optionally, in some embodiments, the third preset value is a value pre-set in the electronic device, and the third preset value is greater than the second preset value. Exemplarily, when the second preset value is 0, the third preset value may be 1.

[0214] When the value of the first element is greater than or equal to the second preset value and less than or equal to the third preset value, the value of the first element remains unchanged, that is, the electronic device does not need to replace the value of the first element. When the value of the first element is greater than the third preset value, the electronic device can replace the value of the first element with the third preset value.

[0215] Optionally, the first element is any element in the detection threshold matrix of the first pixel.

[0216] In summary, the electronic device can perform a permutation process on each element in the detection threshold matrix of the first pixel point according to the permutation process of the first element. After completing the permutation process, the electronic device can obtain the first matrix. The embodiment of the present application can ensure that the range of the elements in the first matrix after the permutation process is not too large by performing a permutation process on one or more elements in the detection threshold matrix of the first pixel point, thereby preparing for the subsequent calculation of the first weight.

[0217] It should be noted that the above replacement process is for each element in the detection threshold matrix of the first pixel point, but after the above replacement process, not every element in the detection threshold matrix of the first pixel point will change. For example, when the second preset value is 0, the third preset value is 1, and the pixel set of the detection threshold matrix of the first pixel point is as follows Fig.11 When the pixel set shown in FIG. 1 is a pixel set of the first matrix, the pixel set obtained by the electronic device is as follows: Fig.12 The pixel set shown, Fig.12 The first matrix shown is Fig.11 In the detection threshold matrix of the first pixel shown, there are two elements with different values.

[0218] Optionally, in some embodiments, after obtaining the first matrix, the electronic device may determine the first weight based on the elements in the first matrix and the first quantity. Exemplarily, the expression of the first weight is as follows:

[0219] ;

[0220] in, represents the first weight; A represents the first matrix; represents the sum of all elements in the first matrix; Indicates the first quantity.

[0221] Exemplarily, the pixel set in the first matrix is Fig.12 In the case of the pixel set shown, .exist hour, .

[0222] In summary, the electronic device can obtain the first weight, and by determining the first weight, it can provide a data basis for the subsequent calculation of the second weight.

[0223] S202, calculating the difference between the first preset value and the first weight to obtain a second weight.

[0224] Optionally, in some embodiments, the second weight is used to represent the weight ratio corresponding to the original pixel value of the first pixel. Exemplarily, the value range of the second weight (β) is (0, 1). The specific value of the second weight may be 0.24.

[0225] Optionally, in some embodiments, the first preset value is a value pre-set in the electronic device. Exemplarily, the first preset value may be 1.

[0226] After obtaining the first weight, the electronic device can obtain the second weight by calculating the difference between the first preset value and the first weight. For example, when the first preset value is 1 and the first weight is 0.76, the second weight is 0.24.

[0227] S203: Determine a target pixel value of the first pixel based on the pixel median of the first pixel, the first weight, the original pixel value of the first pixel, and the second weight.

[0228] Optionally, in some embodiments, the expression of the target pixel value of the first pixel is as follows:

[0229] ;

[0230] in, represents the target pixel value of the first pixel; represents the first weight; represents the pixel median value of the first pixel; represents the second weight; Indicates the original pixel value of the first pixel.

[0231] For example, in is 0.76, is 0.24, is 377.8, and When it is 545, At this time, the pixel set of the first window is as follows Fig.13 The pixel set shown in FIG. 1 has an original pixel value of the first pixel point. Figure 8 The 545 shown is adjusted as follows Fig.13 The target pixel value of the first pixel point shown is 418.

[0232] It should be noted that when the first preset value is 1, It can also be expressed as , the embodiments of the present application do not limit this.

[0233] After obtaining the first weight and the second weight, the electronic device can determine the target pixel value of the first pixel based on the pixel median of the first pixel, the first weight, the original pixel value of the first pixel and the second weight, thereby achieving the effect of bad pixel correction.

[0234] In summary, the electronic device determines the detection threshold matrix of the first pixel by combining the directional gradient information of the first pixel. The embodiment of the present application can improve the success rate of bad pixel detection in the original image data by using the detection threshold matrix of each pixel in the original image data as a basis for judging whether the pixel is a bad pixel. Then, the electronic device determines the first weight based on the detection threshold matrix of the first pixel. Further, the electronic device determines the second weight based on the first weight. Furthermore, the electronic device can obtain the target pixel value of the first pixel, that is, the actual pixel value of the first pixel, by fusing the pixel median of the first pixel and the original pixel value of the first pixel through the first weight and the second weight. When a pixel is a bad pixel, the embodiment of the present application determines the fusion weight ratio (i.e., the first weight) of the pixel median of the first pixel through the detection threshold matrix of the first pixel, and determines the fusion weight ratio (i.e., the second weight) of the original pixel value of the first pixel through the fusion weight ratio of the pixel median of the first pixel, and further, by fusing the obtained weight ratio (including the first weight and the second weight) with the pixel median of the pixel and the original pixel value of the pixel respectively, the target pixel value of the pixel can be made more accurate (i.e., the difference between the target pixel value of the pixel and the original pixel values ​​of other pixels in the neighborhood of the pixel is smaller), and the effect of bad pixel removal is better.

[0235] The above describes in detail the specific implementation process of the image processing method of the embodiment of the present application. In practical applications, the image processing method of the embodiment of the present application can be implemented by relying on the hardware system and software system of the electronic device. Fig.14 and Fig.15 , respectively introduce the software system and hardware system of an electronic device in an embodiment of the present application.

[0236] The software system of the electronic device in the embodiment of the present application may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture or a cloud architecture. The embodiment of the present application takes the layered architecture as an example to exemplarily describe the software system of the electronic device.

[0237] See also Fig.14 , Fig.14 A schematic diagram of a software system of an electronic device provided in an embodiment of the present application is shown.

[0238] like Fig.14 As shown, the software system in the electronic device can be divided into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Fig.14 As shown in the figure, the software system of the electronic device is composed of the application layer (application, APP), application framework layer (applicationframework), hardware abstraction layer (hardware abstraction layer, HAL), driver layer (driver) and hardware layer from top to bottom.

[0239] The application layer may include multiple applications. For example, the multiple applications may be a camera application, a settings application, an album application, a calendar application, and a music application. It is understood that the multiple applications in the application layer may be third-party applications installed by the user or system applications. Figure 1 The camera 101 shown in (1) is used to detect and respond to the interface (e.g., Figure 1 User operations occurring in the second interface 11) shown in (1).

[0240] The application framework layer is a framework layer used to support the operation of multiple applications in the application layer. Fig.14 As shown, the application framework layer includes standard Android application programming interfaces (APIs), and the multiple interfaces can be called by multiple applications in the application layer.

[0241] like Fig.14As shown, the application framework layer may also include core services, etc. The core services are used to provide core functions of the software system (eg, data storage, etc.).

[0242] The hardware abstraction layer is an interface layer between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. The language used in the hardware abstraction layer is a language for defining the hardware abstraction layer interface (HALinterface definition language, HIDL). The language in the hardware abstraction layer is used to describe the interface and functions of the hardware components. The HIDL framework is the HAL framework provided by native Android.

[0243] Optionally, the image processing method of the embodiment of the present application is applied to Fig.14 The hardware abstraction layer shown in Figure 1. Fig.14 As shown, the hardware abstraction layer may include a bad pixel detection module, a bad pixel correction module, and an output module.

[0244] The bad pixel detection module is used to detect bad pixels in the original image data collected by the image sensor in the camera. Specifically, the bad pixel detection module is used to determine the detection threshold matrix of each pixel in the original image data, and judge whether the corresponding pixel is a bad pixel based on the detection threshold matrix of each pixel, which can improve the success rate of bad pixel detection.

[0245] The bad pixel correction module is used to correct the bad pixels in the original image data detected by the bad pixel detection module. Specifically, when the bad pixel detection module determines that a pixel in the original image data is a bad pixel, the bad pixel correction module is used to adjust the original pixel value of the pixel to the target pixel value of the pixel based on the detection threshold matrix of the pixel, so that the difference between the target pixel value of the pixel and the original pixel values ​​of other pixels in the neighborhood of the pixel is small, thereby achieving the purpose of bad pixel correction.

[0246] The output module is used to output a first image with higher definition, more complete image details and better quality according to the target pixel value of each pixel in the original image data obtained by the bad pixel correction module (for example, Figure 1 The black image after removing bad pixels as shown in (2) in the figure).

[0247] like Fig.14 As shown in Figure 1, the driver layer is an abstract layer between hardware and software. The driver layer includes drivers corresponding to various hardware. Fig.14 The camera driver, display driver, hardware interface driver and sensor driver shown.

[0248] The camera driver is used to drive the image sensors of one or more cameras in the camera module to obtain raw image data.

[0249] Optionally, the display driver is used to ensure that the electronic device can display an image after bad pixel correction is performed on the original image data, that is, the first image output by the output module in the application framework layer.

[0250] Optionally, the hardware interface driver includes a hardware connection driver between the Bluetooth chip and the processor. Specifically, the hardware interface driver may be a universal serial bus (USB) interface driver. Alternatively, the hardware interface driver may also be a universal asynchronous receiver / transmitter (UART) interface driver. Thus, the electronic device can be connected to the corresponding hardware through different hardware driver interfaces.

[0251] Optionally, the sensor driver is used to drive different sensors to perform detection and obtain corresponding information.

[0252] Optionally, the hardware layer may include one or more cameras, optical image stabilizers, and touch sensors. Each camera includes an image sensor. An image sensor is a device that converts light signals into electrical signals. The image sensor is used to obtain raw image data. The touch sensor is used to detect touch operations. In the embodiment of the present application, the touch sensor is used to detect the user's application (such as Figure 1 The camera 101 shown in (1) or the control (such as Figure 1 The trigger operation of the third control 105) shown in (2).

[0253] It should be understood that Fig.14 The hierarchical structure shown does not constitute a specific limitation on the software system of the electronic device. In other embodiments of the present application, the software system of the electronic device may include Fig.14 The layered architecture shown may be more or less than the architecture, and the embodiments of the present application are not limited thereto.

[0254] For example, Fig.14 As shown, the camera application in the application layer receives a user's photo-taking instruction (for example, Figure 1After the user clicks the operation of the third control 105 as shown in (2) in the figure), the photo-taking instruction is transmitted to the application framework layer. The application framework layer transmits the photo-taking instruction to the hardware abstraction layer through the Android application programming interface in the application framework layer. The hardware abstraction layer calls the camera driver in the driver layer through the corresponding interface in the hardware abstraction layer. The camera driver drives the camera in the hardware layer to obtain raw image data. The image sensor in the camera can transmit the acquired raw image data to the hardware abstraction layer through the corresponding driver in the driver layer. After receiving the raw image data, the bad pixel detection module in the hardware abstraction layer can perform bad pixel detection on the raw image data, and transmit the bad pixel detection result to the bad pixel correction module. The bad pixel correction module removes the bad pixels detected in the raw image data according to the bad pixel detection result, obtains a first image, and transmits the first image to the output module. The output module transmits the first image to the application framework layer, and the application framework layer transmits it to the album application of the application layer through the Android application programming interface in the application framework layer. The album application saves the first image (for example, as Figure 1 The black image after removing the bad pixels in the second control 104 shown in (3)).

[0255] The above describes in detail a software system of an electronic device in an embodiment of the present application. Fig.15 , introduces a hardware system of an electronic device in an embodiment of the present application.

[0256] See also Fig.15 , Fig.15 A hardware system schematic diagram of an electronic device provided in an embodiment of the present application is shown.

[0257] like Fig.15As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface, namely a USB interface 130, a charging management module 140, a power management module 141, a battery 142, a first antenna 1, a second antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface, namely a SIM card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L and a bone conduction sensor 180M, etc.

[0258] It should be noted that Fig.15 The structure shown does not constitute a specific limitation on the hardware system of the electronic device 100. In other embodiments of the present application, the hardware system of the electronic device 100 may include: Fig.15 The components shown are more or less than the components shown, or the hardware system of the electronic device 100 may include Fig.15 A combination of some of the components shown, or a hardware system of the electronic device 100 may include Fig.15 Subcomponents of some of the components shown. For example, Fig.15 The illustrated proximity light sensor 180G may be optional. Fig.15 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0259] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Different processing units may be independent devices or integrated devices.

[0260] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0261] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0262] Fig.15 The connection relationship between the modules shown is only a schematic illustration and does not constitute a limitation on the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of multiple connection modes in the above embodiments.

[0263] The electronic device 100 can obtain raw image data through the image sensor in the camera 193 .

[0264] The display screen 194 can be used to display an image (e.g., a first image). The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED (Micro OLED), or a quantum dot light emitting diode (QLED). In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0265] In some embodiments, the expanded state of the display screen 194 can be as follows: Figure 1 Any of the 4 states shown in .

[0266] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0267] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0268] The camera 193 is used to capture a still image (e.g., a first image) or a video. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0269] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0270] The touch sensor 180K is also called a touch control device. The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, which is also called a touch control screen. The touch sensor 180K is used to detect a touch operation acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100 and set at a different position from the display screen 194.

[0271] Optionally, in some embodiments, the processor 110 can start the camera 193, based on which, it can obtain the original image data output by the camera, and provide a data basis for subsequent image processing (including bad pixel detection and bad pixel correction); then, the original image data is expanded to obtain a RAW image after expansion. By expanding the original image data, it can be ensured that there is neighborhood information for the edge pixels of the original image data, thereby preparing for subsequent image processing; further, based on the RAW image after expansion, each pixel in the original image data is traversed according to the following traversal process of the first pixel point (which can be any pixel point in the original image data); when traversing the first pixel point, the directional gradient information of the first pixel point is determined; and according to the directional gradient information of the first pixel point, the detection threshold matrix of the first pixel point is determined, which can provide a judgment matrix for subsequent bad pixel detection. Determine basis; further, according to the detection threshold matrix of the first pixel point, it can be determined whether the first pixel point is a bad pixel; when the first pixel point is a bad pixel point, the original pixel value of the first pixel point is adjusted to the target pixel value of the first pixel point, and the actual pixel value of the first pixel point, that is, the difference between the target pixel value of the first pixel point and the original pixel values ​​of other pixels in the neighborhood of the first pixel point is reduced by adjusting the original pixel value of the first pixel point, so as to improve the similarity between the first pixel point and other pixels in the neighborhood of the first pixel point; based on the above description, after traversing each pixel point in the original image data, the target pixel value of each pixel point in the original image data can be obtained, and each pixel point in the original image data has a high similarity with other pixels in the neighborhood of the corresponding pixel point; finally, according to the target pixel value of each pixel point in the original image data, the first image is output (for example, Figure 1 The black image after removing bad pixels as shown in (2) in the figure can obtain an image with higher definition and better quality.

[0272] In summary, it can be seen that the image processing method of the embodiment of the present application determines the detection threshold matrix of each pixel point by combining the directional gradient information of each pixel point, and uses the detection threshold matrix of each pixel point as the basis for judging whether the pixel point is a bad pixel, which can improve the success rate of bad pixel detection. In addition, when a pixel point is determined to be a bad pixel point, the original pixel value of the pixel point can be adjusted to obtain a more accurate target pixel value of the pixel point, so that the difference between the target pixel value of the pixel point and the original pixel value of other pixels in the neighborhood of the pixel point is small, and the similarity between the pixel point and other pixels in the neighborhood of the pixel point is improved, so as to achieve the effect of removing bad pixels, and the image information of the output first image is more complete, and the image is clearer and more delicate.

[0273] Illustratively, the present application provides a readable storage medium, in which codes or instructions are stored. The processor 110 calls the instructions so that the electronic device 100 implements the image processing method in the foregoing embodiment when executing the instructions.

[0274] Exemplarily, the present application provides a chip system, which is applied to an electronic device 100 including a memory, a display screen 194 and a sensor; the chip system includes: one or more interface circuits and one or more processors 110; the interface circuit and the processor 110 are interconnected through lines; the interface circuit is used to receive a signal from the memory and send a signal to the processor 110, the signal including a computer code or instruction stored in the memory; the processor 110 calls the instruction so that the electronic device 100 executes the image processing method in the previous embodiment.

[0275] Exemplarily, the present application provides a computer program product, including a computer program. When the computer program product runs on a computer, the computer executes the image processing method in the foregoing embodiment.

[0276] In the above embodiments, all or part of the functions can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer codes or instructions. When the computer program code or instructions are loaded and executed on the computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer code or instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0277] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. An image processing method, characterized in that: The method comprises: Start the camera; Obtaining raw image data output by the camera; Performing expansion processing on the original image data to obtain an expanded RAW image; Based on the RAW image after the expansion process, traverse each pixel point in the original image data; When traversing a first pixel point, determining directional gradient information of the first pixel point, where the first pixel point is any pixel point in the original image data; Determining a detection threshold matrix of the first pixel point according to the directional gradient information of the first pixel point; Determining whether the first pixel is a bad pixel according to a detection threshold matrix of the first pixel; When the first pixel is a bad pixel, adjusting the original pixel value of the first pixel to the target pixel value of the first pixel; After traversing each pixel point in the original image data, obtaining a target pixel value for each pixel point in the original image data; A first image is output according to the target pixel value of each pixel in the original image data.

2. The method according to claim 1, characterized in that: When the first pixel is a bad pixel, adjusting the original pixel value of the first pixel to a target pixel value includes: Determine a target pixel value of the first pixel based on a pixel median value of the first pixel and an original pixel value of the first pixel; the pixel median value of the first pixel is the median of the original pixel values ​​of all pixels in the detection area corresponding to the first pixel; The original pixel value of the first pixel point is adjusted to the target pixel value of the first pixel point.

3. The method according to claim 2, characterized in that The determining the target pixel value of the first pixel point based on the pixel median value of the first pixel point and the original pixel value of the first pixel point includes: Based on the pixel median, the first weight, the original pixel value of the first pixel and the second weight of the first pixel, the target pixel value of the first pixel is determined; the first weight is used to represent the weight ratio corresponding to the pixel median of the first pixel; the second weight is used to represent the weight ratio corresponding to the original pixel value of the first pixel.

4. The method according to claim 1, characterized in that: The determining whether the first pixel is a bad pixel according to the detection threshold matrix of the first pixel includes: Performing a replacement process on one or more elements in the detection threshold matrix of the first pixel point to obtain a first matrix; Based on the first matrix, determining a first number; the first number is the number of elements in the first matrix that are greater than a second preset value; When the first number is greater than or equal to a first threshold, determining that the first pixel is a bad pixel; When the first number is smaller than the first threshold, it is determined that the first pixel is not a bad pixel.

5. The method according to claim 4, characterized in that The performing permutation processing on one or more elements in the detection threshold matrix of the first pixel point to obtain a first matrix includes: Based on the detection threshold matrix of the first pixel point, each element in the detection threshold matrix of the first pixel point is replaced according to the following first element replacement method to obtain the first matrix: When the value of the first element is less than the second preset value, the value of the first element is replaced by the second preset value; When the value of the first element is greater than or equal to the second preset value and less than or equal to the third preset value, the value of the first element remains unchanged; When the value of the first element is greater than the third preset value, replacing the value of the first element with the third preset value; The first element is any element in the detection threshold matrix of the first pixel point.

6. The method according to claim 4, characterized in that In the case where it is determined that the first pixel is a bad pixel, the method further includes: After obtaining the first matrix, a first weight is determined based on the elements in the first matrix and the first quantity.

7. The method according to claim 6, characterized in that The second weight is the difference between the first preset value and the first weight.

8. The method according to claim 1, characterized in that: The traversing each pixel point in the original image data based on the RAW image after the expansion processing includes: Based on the RAW image after the expansion process, a first window is determined; pixels in the first window are located in the RAW image after the expansion process; Based on the first window, traverse each pixel point in the original image data; When traversing the first pixel point, the first window is a detection area centered on the first pixel point.

9. The method according to claim 8, characterized in that The method further comprises: Arrange the original pixel values ​​of the pixels in the first window in sequence to obtain a first sequence; The median of the first sequence is determined as the pixel median of the first pixel point.

10. The method according to claim 8, characterized in that The determining the directional gradient information of the first pixel point includes: The difference between the original pixel value of each pixel in the first window and the original pixel value of the first pixel is calculated respectively to obtain the directional gradient information of the first pixel.

11. The method according to claim 1, characterized in that The step of determining a detection threshold matrix of the first pixel point according to the directional gradient information of the first pixel point includes: A detection threshold matrix of the first pixel is determined according to the directional gradient information of the first pixel, a first parameter and a second matrix; the first parameter is a coefficient of the directional gradient information of the first pixel; and the second matrix is ​​determined based on the second parameter and a unit matrix.

12. The method according to claim 11, characterized in that The value of the first parameter is positively correlated with the probability of determining whether the first pixel is a bad pixel, and the value of the second parameter is negatively correlated with the probability of determining whether the first pixel is a bad pixel.

13. The method according to claim 1, characterized in that The starting the camera includes: In response to a first operation of the user, starting the camera; The outputting the first image comprises: In response to a photographing operation of the user, the first image is output.

14. The method according to claim 1, characterized in that In the case where the first pixel is not a bad pixel, the method further includes: The original pixel value of the first pixel point is used as the target pixel value of the first pixel point.

15. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor and the memory are coupled, the memory is used to store a computer program, and the processor calls instructions so that the electronic device executes the method according to any one of claims 1 to 14.

16. A computer program product, characterized in that The invention comprises a computer program which, when executed, causes the method according to any one of claims 1 to 14 to be performed.

17. A chip system, characterized in that: The chip system is applied to an electronic device including a memory, a display screen and a sensor; the chip system includes: one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive signals from the memory and send signals to the processor, and the signals include computer codes or instructions stored in the memory; the processor calls instructions so that the electronic device executes the method described in any one of claims 1 to 14.

Images