Image processing method, electronic device, storage medium and computer program product

Abstract

The application provides an image processing method, an electronic device, a storage medium and a computer program product. The method comprises: obtaining a to-be-processed image; identifying a defect region from the image, the initial color space corresponding to the image having one or more color channels, which correspond to the same defect positioning result or one-to-one correspond to one or more defect positioning results, for any pixel, if there are a target number of defect positioning results indicating that the pixel belongs to a defect category, the pixel is a comprehensive defect pixel; for each of at least part of the comprehensive defect pixels, performing a pixel replacement operation: for each defect positioning result indicating that the comprehensive defect pixel belongs to the defect category, based on the result, finding a set of target pixels located around the comprehensive defect pixel and satisfying a target requirement; calculating a new pixel value based on at least the pixel values of the found target pixels; and replacing the original pixel value of the comprehensive defect pixel with the new pixel value. The defects can be efficiently and accurately removed.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically to an image processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] Users typically perform post-processing on images or videos after shooting them for aesthetic purposes. With advancements in photographic technology, both hardware and software, various image post-processing techniques have emerged. Imperfection removal is one such technique. Imperfections refer to elements on the face or other parts of the body that affect appearance, such as pimples, blemishes, stray hairs, and redness in the eyes—all of which need to be removed.

[0003] Currently, there is a lack of efficient blemish removal technology in the field of image processing. Summary of the Invention

[0004] This application is made in view of the above-mentioned problems. This application provides an image processing method, an electronic device, a storage medium, and a computer program product.

[0005] According to one aspect of this application, an image processing method is provided, comprising: acquiring an image to be processed, the image to be processed including a human skin region; identifying defect regions from the image to be processed, wherein the initial color space corresponding to the image to be processed has one or more color channels, the one or more color channels corresponding to the same defect localization result, or the one or more color channels corresponding one-to-one with one or more defect localization results, wherein for any pixel on the image to be processed, if there is a target number of defect localization results indicating that the pixel belongs to a defect class, then the pixel is a composite defect pixel, the defect region includes at least a portion of the composite defect pixels in the image to be processed, and the target number is greater than or equal to 1 and less than or equal to one or more. The total number of defect localization results; for each composite defect pixel in at least a portion of composite defect pixels in the defect region, perform the following pixel replacement operation: for each defect localization result indicating that the composite defect pixel belongs to a defect class, based on the defect localization result, find a set of target pixels around the composite defect pixel that meet the target requirements, wherein the set of target pixels that meet the target requirements includes at least one channel non-defect pixel, the channel non-defect pixel being a pixel that the defect localization result indicates belongs to a non-defect class; calculate a new pixel value for the composite defect pixel based at least on the pixel value of the found target pixels; replace the original pixel value of the composite defect pixel on the image to be processed with the new pixel value of the composite defect pixel.

[0006] According to another aspect of this application, an electronic device is provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the above-described image processing method.

[0007] According to another aspect of this application, a storage medium is provided on which program instructions are stored, which, when run, are used to execute the above-described image processing method.

[0008] According to another aspect of this application, a computer program product is provided, the computer program product comprising a computer program, which, when running, is used to perform the above-described image processing method.

[0009] According to the image processing method, electronic device, storage medium, and computer program product of the embodiments of this application, an independent and consistent pixel replacement operation can be performed for each defective pixel. In this way, the operations of different defective pixels are independent of each other and do not need to adhere to strict timing relationships. That is, the pixel replacement operation of each defective pixel can be performed independently as needed. This scheme of independently executing pixel replacement operations helps to improve the efficiency of defect removal. Furthermore, thanks to this independent execution scheme, different defective pixels do not interfere with each other, and pixel replacement errors do not accumulate between pixels; therefore, this scheme also helps to improve the accuracy of defect removal. The image processing method according to the embodiments of this application is an efficient and accurate method for removing defects. Attached Figure Description

[0010] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 A schematic block diagram of an example electronic device for implementing the image processing method and apparatus according to embodiments of this application is shown;

[0012] Figure 2a A schematic flowchart illustrating an image processing method according to an embodiment of this application is shown;

[0013] Figure 2b A schematic flowchart illustrating a pixel replacement operation performed for each composite defective pixel according to an embodiment of this application;

[0014] Figure 3A schematic diagram showing any composite defect pixel and a plurality of surrounding pixels on an image to be processed according to an embodiment of the present application;

[0015] Figure 4 A schematic block diagram of an image processing apparatus according to an embodiment of the present application is shown; and

[0016] Figure 5 A schematic block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0017] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include facial recognition, image processing, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beauty filters, cosmetics, medical aesthetics, and intelligent temperature measurement.

[0018] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0019] This application provides an image processing method, an electronic device, a storage medium, and a computer program product. The image processing method according to this application allows for independent yet consistent pixel replacement operations for each pixel with overall defects, facilitating efficient defect removal. The image processing technology according to this application can be applied to any field involving image or video applications, including but not limited to various image or video capture, live streaming, facial recognition, and identity authentication.

[0020] First, refer to Figure 1 This describes an example electronic device 100 for implementing the image processing method and apparatus according to embodiments of this application.

[0021] like Figure 1 As shown, the electronic device 100 includes one or more processors 102 and one or more storage devices 104. Optionally, the electronic device 100 may also include an input device 106, an output device 108, and an image acquisition device 110, these components being interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device may also have other components and structures as needed.

[0022] The processor 102 may be implemented in at least one of the following hardware forms: digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic array (PLA), and microprocessor. The processor 102 may be one or a combination of several of the following: central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), or other processing units with data processing capabilities and / or instruction execution capabilities. It may also control other components in the electronic device 100 to perform the desired functions.

[0023] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of this application described below, and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0024] The input device 106 may be a device used by a user to input commands, and may include one or more of the following: keyboard, mouse, microphone, and touch screen.

[0025] The output device 108 can output various information (e.g., images and / or sound) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. Optionally, the input device 106 and the output device 108 can be integrated together and implemented using the same interactive device (e.g., a touch screen).

[0026] The image acquisition device 110 can acquire images and store the acquired images in the storage device 104 for use by other components. The image acquisition device 110 can be a standalone camera or a camera in a mobile terminal, etc. It should be understood that the image acquisition device 110 is only an example, and the electronic device 100 may not include the image acquisition device 110. In this case, other devices with image acquisition capabilities can be used to acquire images and send the acquired images to the electronic device 100.

[0027] For example, an example electronic device for implementing the image processing method and apparatus according to embodiments of this application may be implemented on a device such as a personal computer or a remote server.

[0028] Below, we will refer to Figure 2a and 2b An image processing method according to an embodiment of this application is described. Figure 2a A schematic flowchart of an image processing method 200 according to an embodiment of this application is shown. Figure 2a As shown, the image processing method 200 includes steps S210, S220 and S230.

[0029] In step S210, an image to be processed is obtained, which includes the human skin region.

[0030] The image to be processed can be a still image or any frame from a moving video. It can be the original image captured by an image acquisition device (such as the image sensor in a camera), or an image obtained after preprocessing the original image (such as digitization, normalization, smoothing, etc.). Note that preprocessing the original image can include extracting a sub-image containing the target object from the original image captured by the image acquisition device to obtain the image to be processed. The target object can be a human skin area, including but not limited to the face, limbs, torso, etc. Defect removal can refer to removing blemishes from the human skin area.

[0031] In step S220, defect regions are identified from the image to be processed. The initial color space corresponding to the image to be processed has one or more color channels. One or more color channels correspond to the same defect location result, or one or more color channels correspond one-to-one with one or more defect location results. For any pixel in the image to be processed, if there is a target number of defect location results indicating that the pixel belongs to the defect category, then the pixel is a composite defect pixel. The defect region contains at least some composite defect pixels in the image to be processed. The target number is greater than or equal to 1 and less than or equal to the total number of one or more defect location results.

[0032] In one example, the initial color space corresponding to the image to be processed has one channel, such as a grayscale image. In this case, the image to be processed may only have one defect localization result. Based directly on this defect localization result, it is possible to uniquely determine which pixels in the image to be processed belong to the defect class and which pixels belong to the non-defect class, thereby determining which or which regions in the image to be processed are defective.

[0033] In another example, the initial color space corresponding to the image to be processed has multiple channels, such as an RGB image. In this case, the identification of defective regions in the image to be processed can be further divided into two situations. One is that all color channels correspond to the same defect localization result. Similar to the case with only one channel, the image to be processed has only one defect localization result, so it is possible to directly and uniquely determine which pixels in the image to be processed belong to the defective class and which pixels belong to the non-defective class based on this defect localization result. The other is that multiple color channels correspond one-to-one with multiple defect localization results. In this case, since different color channels have different defect localization results, the same pixel may have different results for different channels regarding whether it belongs to the defective or non-defective class. In this case, it can be defined that if there are a target number (which can be called the first target number) of defect localization results indicating that a certain pixel belongs to the defective class, then the pixel is determined to belong to the defective class (for ease of distinction, this paper refers to such pixels that are identified as belonging to the defective class after combining various defect localization results as comprehensive defective pixels). The target number can be set to any suitable value as needed, such as 1, 2, 3, etc. For example, assuming the image to be processed is an RGB image, for the pixel at coordinates (10, 20), the defect location result corresponding to the R channel indicates that the pixel is defective, while the defect location results corresponding to the B and G channels both indicate that the pixel is not defective. Assuming the target number is 1, then the pixel at (10, 20) can be considered a composite defect pixel.

[0034] Step S220 can be implemented using any existing or future defect identification method. Exemplarily, and not limitingly, it can be combined with certain image enhancement algorithms, such as Limit Contrast Adaptive Histogram Equalization (CLAHE) or high-frequency separation algorithms, to identify defect regions from the image to be processed. Exemplary defect identification methods will be described below.

[0035] The number of image regions included in the defective region identified in step S220 can be one or more. When multiple image regions are identified as defective regions, the pixels within each image region are connected to each other in position, while different image regions can be independent of each other and not connected to each other.

[0036] In step S230, for each composite defect pixel in at least a portion of the composite defect pixels in the defect region, a pixel replacement operation is performed.

[0037] In one example, pixel replacement can be performed on every single composite defect pixel within a defective region. In another example, the composite defect pixels in the defective region can be further filtered, performing pixel replacement only on a subset of those composite defect pixels. The filtering method can be set as needed. For example, for the same image region, composite defect pixels at the edges of the image region can be ignored, and pixel replacement can be performed only on the remaining composite defect pixels near the center.

[0038] For ease of description, we assume that the number of at least some composite defective pixels in the defective region is N, where N is a positive integer greater than or equal to 1. The following describes the implementation of the pixel replacement operation for the i-th composite defective pixel among the N composite defective pixels, where i = 1, 2, 3, ..., N.

[0039] Figure 2b A schematic flowchart illustrating a pixel replacement operation performed for each composite defective pixel according to an embodiment of this application is shown. Figure 2b As shown, the pixel replacement operation includes the following steps S232, S234 and S236.

[0040] In step S232, for each defect location result indicating that the composite defect pixel (the i-th composite defect pixel) belongs to the defect class, based on the defect location result, a set of target pixels that meet the target requirements are found around the composite defect pixel. The set of target pixels that meet the target requirements includes at least one channel non-defect pixel, which is a pixel that the defect location result indicates belongs to the non-defect class.

[0041] For any color channel, if the defect localization result corresponding to that color channel indicates that any pixel in the image to be processed belongs to the defect category, then that pixel belongs to the channel defect pixel corresponding to that color channel. Conversely, if the defect localization result corresponding to that color channel indicates that any pixel in the image to be processed belongs to the non-defect category, then that pixel belongs to the channel non-defect pixel corresponding to that color channel. For ease of description, the color channel corresponding to the defect localization result indicating that any pixel belongs to the defect category can be called the defect channel. It can be understood that for any pixel in the image to be processed, if the defect localization results indicating that it belongs to the defect category (i.e., indicating that it is a channel defect pixel) reach the target number, then that pixel is a comprehensive defect pixel.

[0042] The target pixel search operation is performed for each defect localization result (whose corresponding color channel is a defect channel) that indicates the overall defect pixel belongs to the defect category. It's understandable that if one or more color channels correspond to the same defect localization result, and the pixel is an overall defect pixel, then all its color channels are defect channels. In this case, the target pixel found is the same for all defect channels. However, if one or more color channels correspond one-to-one with one or more defect localization results, and the number of color channels is multiple, then if the pixel is an overall defect pixel, at least some of its color channels are defect channels, and the rest are normal color channels (the number of normal color channels may be 0). In this case, the target pixel search operation can be performed independently for different defect channels of the overall defect pixel; that is, the target pixels found for different defect channels may be the same or different. For example, assuming that for the i-th overall defect pixel, both the R channel and the B channel are defect channels, then the target pixel found for the R channel and the target pixel found for the B channel can be exactly the same, completely different, or partially the same and partially different.

[0043] The pixels surrounding the i-th composite defective pixel can be understood as pixels that surround the i-th composite defective pixel in a 360° radius. For example, all pixels surrounding the i-th composite defective pixel that are one checkerboard square away from it can be considered as the search range. Those skilled in the art will understand that these pixels are the 8-neighborhood pixels of the i-th composite defective pixel. If these 8-neighborhood pixels meet the target requirements, they are taken as the target pixels.

[0044] The target requirements can be set as needed. For example, it can be set to require that the number of channel-free pixels in a set of target pixels be greater than a certain number, or that the proportion of channel-free pixels in a set of target pixels be greater than a certain proportion, and so on.

[0045] A set of target pixels that meets the target requirements will not consist entirely of channel-defective pixels; that is, it will include at least one channel-free pixel. Therefore, at least one channel-free pixel from the set of target pixels that meets the target requirements can be used to replace i composite defective pixels.

[0046] In step S234, a new pixel value for the composite defective pixel is calculated based at least on the pixel value of the found target pixel.

[0047] The target pixels found in step S234 can be the union of one or more sets of target pixels that correspond one-to-one with one or more defect channels of the i-th integrated defect pixel. As mentioned above, the target pixels corresponding to each defect channel may be the same or different. For ease of distinction, the target pixels corresponding to each defect channel (referred to as a set of target pixels) are described separately. When the number of defect channels is one or more, and each defect channel corresponds to the same defect localization result, the found target pixels are the same set of target pixels. When the number of defect channels is multiple, and each corresponds to a set of target pixels, the found target pixels may include the independent pixels of each defect channel as well as pixels that overlap with each other. For example, suppose that for the i-th composite defective pixel, both the R channel and the B channel are defective channels, and the target pixels found for the R channel are the first group of target pixels and the target pixels found for the B channel are the second group of target pixels. Also suppose that the first group of target pixels has 10 pixels and the second group of target pixels has 20 pixels, and there are 4 overlapping pixels between the two. Then the target pixels found in step S234 may include the remaining 6 pixels in the first group of target pixels, the remaining 16 pixels in the second group of target pixels, and the 4 overlapping pixels.

[0048] In one example, the new pixel value of the i-th composite defect pixel can be calculated solely based on the pixel value of the found target pixel. In another example, the new pixel value of the i-th composite defect pixel can be calculated by combining the original pixel value of the i-th composite defect pixel with the pixel value of the found target pixel.

[0049] In step S236, the original pixel value of the composite defect pixel on the image to be processed is replaced with the new pixel value of the composite defect pixel.

[0050] The result of performing image processing method 100 can be a de-defected image. Pixels in the image to be processed that are located outside the defective region can be referred to as composite non-defective pixels. For any composite non-defective pixel in the image to be processed, its original pixel value can be directly used as the pixel value of the pixel at the same position in the de-defected image. For any composite defective pixel in the image to be processed that does not participate in the pixel replacement operation, it can be processed in a similar manner to the composite non-defective pixels described above.

[0051] For the i-th composite defect pixel among N composite defect pixels, the original pixel value of that composite defect pixel in the image to be processed can be replaced with the calculated new pixel value. In other words, for the i-th composite defect pixel, its new pixel value can be used as the pixel value of the pixel at the same position as the i-th composite defect pixel in the de-defect image.

[0052] The above processing method yields a blemish-free image. In this blemish-free image, at least a portion of the pixel values ​​of the blemish pixels within the original blemish area are suppressed. This is achieved by balancing the pixel values ​​using surrounding non-blemish pixels, reducing the contrast between the previously prominent blemish (e.g., too dark or too bright) and its surroundings. Thus, visually, the blemish area is softened.

[0053] According to embodiments of this application, an independent and consistent pixel replacement operation can be performed for each defective pixel. In this way, the operations on different defective pixels are independent of each other and do not need to adhere to strict timing relationships. That is, the pixel replacement operation for each defective pixel can be performed independently as needed. This scheme of independently executing pixel replacement operations helps improve the efficiency of defect removal. Furthermore, thanks to this independent execution scheme, different defective pixels do not interfere with each other, and pixel replacement errors do not accumulate between pixels; therefore, this scheme also helps improve the accuracy of defect removal. In summary, the image processing method according to embodiments of this application is an efficient and accurate method for removing defects.

[0054] For example, the image processing method according to the embodiments of this application can be implemented in a device, apparatus or system having a memory and a processor.

[0055] The image processing method according to the embodiments of this application can be deployed at the image acquisition end, for example, at a personal terminal or a server.

[0056] Alternatively, the image processing method according to embodiments of this application can also be deployed distributedly on a server (or cloud) and a personal terminal. For example, a raw image or an image to be processed can be captured on a client side, and the client can transmit the captured image to the server (or cloud) side for image processing.

[0057] According to an embodiment of this application, for each defect location result indicating that the overall defect pixel belongs to a defect category, finding a set of target pixels that meet the target requirements around the overall defect pixel based on the defect location result includes:

[0058] For each defect localization result indicating that the overall defect pixel belongs to the defect class, perform the following lookup operation:

[0059] Step a: Based on the defect location result, determine whether the pixels in the current search range meet the target requirements. If they do, proceed to step b; otherwise, proceed to step c.

[0060] Step b: Determine a set of target pixels within the current search range that meet the target requirements and end the search;

[0061] Step c: Determine a new range around the composite defective pixel based on the current search range. The inner boundary of the new range coincides with the outer boundary of the current search range, or the inner boundary of the new range is further out than the outer boundary of the current search range.

[0062] Step d: Determine the new range as the current search range and return to step a.

[0063] It is understood that the ranges described in this article (e.g., the initial range, the new range, and the current search range) are all around the composite defective pixel; that is, the interior of the range can be 360° connected. It should be noted that connectivity in this article means that pixels within the range are positionally adjacent to each other, not that their pixel values ​​are identical. The range described in this article can be any suitable range, and its size can be arbitrarily set. In one example, the range described in this article can refer to an annular (approximately annular) range of pixels on the image to be processed, centered on the i-th composite defective pixel. In another example, the range described in this article can refer to the range covered by pixels that are a certain Euclidean distance or chessboard distance from the i-th composite defective pixel.

[0064] When searching for a set of target pixels that meet the target requirements, the search can begin from the inside out. For example, an initial range can be defined around the i-th defective pixel, and the search can start from this initial range (which is considered the current search range). If any pixels within this range meet the target requirements, then the pixels within this initial range are directly identified as a set of target pixels, and the search ends. Conversely, if the pixels within the initial range do not meet the target requirements, a new range further out than the initial range can be defined, and the pixels within this new range can be checked to see if they meet the target requirements (this new range is considered the current search range). This search process can be repeated. Each time a pixel within the current search range does not meet the target requirements, the search continues outwards to check if the pixels in the next range meet the target requirements, until a range that meets the target requirements is found. Finally, the pixels within the range that meet the target requirements are considered a set of target pixels.

[0065] Optionally, the current search range can be limited. For example, in step b, it can be further determined whether the current search range is greater than the target range. For instance, in an embodiment where the current search range is adjusted in steps of a certain chessboard distance, it can be determined whether the chessboard distance corresponding to the current search range is greater than the target chessboard distance. If so, the search can be terminated directly. This approach avoids the target pixel being too far from the overall defect pixel, thus affecting the defect removal effect.

[0066] By using the above method, we can systematically search for neighboring pixels that can replace the defective pixels from the inside out. That is, we try to find pixels that are closer to the defective pixels for pixel replacement. This helps to ensure the consistency between the defective pixels and the neighboring pixels, resulting in a better defect removal effect.

[0067] Furthermore, by using the above method, for each pixel with a comprehensive defect, it is only necessary to traverse its surrounding pixels once (i.e., there is no need to repeatedly search from the same pixels). If the pixels in the current search range can meet the target requirements as soon as possible, the search can also end as soon as possible.

[0068] According to an embodiment of this application, the current search range is the range covered by pixels that are a first chessboard distance away from the defective pixel, and the new range is the range covered by pixels that are a second chessboard distance away from the defective pixel, where the second chessboard distance is greater than the first chessboard distance.

[0069] The initial range is the area covered by pixels that are at an initial chessboard distance from the defective pixel. It can be understood that, given the current search range is the initial range, the first chessboard distance is the initial chessboard distance.

[0070] Figure 3A schematic diagram is shown illustrating any composite defect pixel and a plurality of surrounding pixels on an image to be processed according to an embodiment of this application. The following is in conjunction with... Figure 3 Describe how to determine the current search range for combined defective pixels.

[0071] See Figure 3 This shows any composite defect pixel S and pixels located one, two, and three squares away from it, respectively. Figure 3 In this game, pixels at different distances from each other on the chessboard are represented by different background patterns.

[0072] Initially, the search range can be set to the pixels one checkerboard square away from the defective pixel S (i.e., its 8 neighboring pixels). It can be determined whether these 8 pixels meet the target requirement. If not, the search range can be expanded to the pixels two checkerboard squares away from the defective pixel S, and the 16 pixels within that range can be checked. If still not, the search range can be expanded to the pixels three checkerboard squares away from the defective pixel S, and the 24 pixels within that range can be checked. If still not, a new range can be defined, and this process can be repeated until a set of target pixels that meets the target requirement is obtained.

[0073] The method of adjusting the current search range by a certain chessboard distance as the step size to find the target pixel is simple to calculate, has high search efficiency, and is easy to implement.

[0074] According to an embodiment of this application, the second chessboard distance differs from the first chessboard distance by one or more chessboard squares, where each chessboard square is one pixel. The second chessboard distance and the first chessboard distance can differ by a target number of pixels, where the target number can be one or more.

[0075] In one example, the distance to the second chessboard differs from the distance to the first chessboard by one pixel. In this case, each new range increases by one chessboard square compared to the current search range. This can be referenced in the above combination. Figure 3 The described embodiments are for understanding. However, the above embodiments are merely examples and not limitations of this application. The new range can differ from the current search range by a greater distance. For example, the second chessboard distance can differ from the first chessboard distance by two pixels. That is, assuming the initial range is the range covered by pixels one chessboard square away from the i-th composite defective pixel, then the second search range could be the range covered by pixels three chessboard squares away from the defective pixel, the third search range could be the range covered by pixels five chessboard squares away from the defective pixel, and so on.

[0076] It's understandable that when the distance to the second chessboard differs from the distance to the first chessboard by one chessboard square, the inner boundary of the new range coincides with the outer boundary of the current search range. When the distance to the second chessboard differs from the distance to the first chessboard by multiple chessboard squares, the inner boundary of the new range is further outward than the outer boundary of the current search range.

[0077] By setting the distance between the second and first chessboards to differ by several chessboard squares—that is, by setting the inner boundary of the new range to be further out than the outer boundary of the current search range—the search for target pixels can be performed in a skipping, non-continuous manner. Since defects typically occupy a certain area, this skipping search method helps to quickly skip the locations of pixels that are also defects, thus finding the target pixel that meets the requirements as quickly as possible. This improves search efficiency and consequently, image processing efficiency.

[0078] According to the embodiments of this application, the proportion of channel-free pixels in a set of target pixels that meet the target requirements is greater than or equal to a target proportion threshold, and / or the number of channel-free pixels in a set of target pixels that meet the target requirements is greater than or equal to a target number threshold.

[0079] Both the target proportion threshold and the target number threshold can be set as needed, and can be of any size; this application does not impose any restrictions on this. In one example, the proportion is used to determine whether the pixels within the search range meet the requirements and can be considered as a group of target pixels. For example, for any defective channel, it can be required that the proportion of non-defective pixels in the found group of target pixels is not less than 40%, 50%, 80%, 100%, etc. The requirement that the proportion of non-defective pixels in the group of target pixels is not less than 100% means that there cannot be any defective pixels in the group of target pixels. In another example, the number is used to determine whether the pixels within the search range meet the requirements and can be considered as a group of target pixels. For example, for any defective channel, it can be required that the number of non-defective pixels in the found group of target pixels is not less than 1, 3, 5, 8, etc.

[0080] By controlling the proportion or number of channel-free pixels in the target pixel, the target pixel can contain as many channel-free pixels as possible to better suppress the pixel value of the combined defective pixels.

[0081] According to embodiments of this application, the found target pixel is the union of one or more sets of target pixels that correspond one-to-one with one or more defect channels of the composite defect pixel. The defect channel is the color channel that the corresponding defect location result indicates to the composite defect pixel belongs to the defect category.

[0082] Based at least on the pixel value of the found target pixel, the calculation of the new pixel value for the composite defect pixel includes:

[0083] For each defective channel of the composite defective pixel, the defective channel components of at least a portion of the target pixels in the set of target pixels corresponding to the defective channel are averaged to obtain the pixel mean. The at least a portion of the target pixels corresponding to any defective channel include only channel non-defective pixels corresponding to the defective channel or include channel non-defective pixels and channel defective pixels corresponding to the defective channel. The channel defective pixels corresponding to any defective channel are pixels that belong to the defective class as indicated by the defect localization result corresponding to the defective channel.

[0084] The pixel mean and the original pixel value of the combined defective pixel are converted to a new color space that includes a luminance channel to obtain the converted pixel mean and the converted pixel value. The luminance channel is used to represent the brightness of the pixel.

[0085] The luminance channel component in the mean value of the transformed pixel and the luminance channel component in the transformed pixel value are weighted and averaged using the first weighting relationship to obtain the luminance channel component of the new pixel value.

[0086] The non-brightness channel component in the mean value of the transformed pixel and the non-brightness channel component in the transformed pixel value are weighted and averaged using the second weighting relationship to obtain the non-brightness channel component of the new pixel value.

[0087] The new pixel value is obtained based on the luminance channel component and the non-luminance channel component of the new pixel value;

[0088] In the first weighted relationship, the weight corresponding to the luminance channel component in the converted pixel mean is greater than 0, and in the second weighted relationship, the weight corresponding to the non-luminance channel component in the converted pixel mean is equal to or greater than 0.

[0089] The meaning of one or more sets of target pixels corresponding one-to-one with one or more defect channels of the i-th composite defect pixel has been described above, and will not be repeated here.

[0090] In the step of averaging the defective channel components of at least a portion of the target pixels in a set of target pixels corresponding to each defective pixel in the overall defective pixel, at least a portion of the pixels may be pixels of a specific type. The specific type of pixels may include only one type of pixel, namely channel non-defective pixels, or include both types of pixels, namely channel defective pixels and channel non-defective pixels.

[0091] In the first example, the specific type of pixel includes only channel-free defective pixels. In this case, regardless of which pixels are included in a set of target pixels, only channel-free defective pixels can be selected to replace the i-th composite defective pixel. In the second example, the specific type of pixel can include both channel-free defective pixels and channel-free defective pixels. In this case, both channel-free defective pixels and channel-free defective pixels are allowed to replace the i-th composite defective pixel. Of course, it is understandable that although both channel-free defective pixels and channel-free defective pixels are allowed to replace the i-th composite defective pixel, if a set of target pixels only includes channel-free defective pixels, then only channel-free defective pixels will still be used for replacement.

[0092] In both examples above, it is preferable to include only channel-free pixels for a specific pixel type. This ensures that the i-th composite defective pixel is replaced only with channel-free pixels, making the new pixel value of the i-th composite defective pixel as close as possible to the surrounding non-defective pixels, thus achieving a better defect removal effect. Of course, the second example is also feasible. In this case, it is not necessary to distinguish and select pixels from a set of target pixels; the new pixel value of the i-th composite defective pixel can be calculated directly based on all pixels or a randomly selected subset of pixels.

[0093] For example, for each defective channel of the composite defective pixel, averaging the defective channel components of at least a portion of the target pixels in a set of target pixels corresponding to the defective channel to obtain a pixel mean may include: for each defective channel of the composite defective pixel, averaging the defective channel components of at least a portion of the target pixels in a set of target pixels corresponding to the defective channel to obtain an average defective channel component corresponding to the defective channel; obtaining the pixel mean of the composite defective pixel based at least on the average defective channel components that correspond one-to-one with one or more defective channels of the composite defective pixel, wherein, in the pixel mean, the channel component corresponding to any defective channel is equal to the average defective channel component corresponding to the defective channel, and the channel component corresponding to any non-defective channel is equal to the channel component of the composite defective pixel in that non-defective channel.

[0094] It can be understood that for any defective channel, the channel component of any pixel in that defective channel is the defective channel component corresponding to that defective channel. Averaging the defective channel components of a specific type of pixel within a set of target pixels corresponding to any defective channel yields the average defective channel component corresponding to that defective channel. This average defective channel component can be used as the channel component corresponding to that defective channel in the pixel mean. For any other ordinary color channel (which can be called a non-defective channel) of the i-th composite defective pixel that does not belong to a defective channel, the channel component corresponding to that non-defective channel can be used as the channel component of the pixel mean in that non-defective channel. This is how the pixel mean is obtained.

[0095] For example, suppose that for the i-th composite defective pixel, both the R and B channels are defective channels, and the G channel is a non-defective channel. Furthermore, suppose that the R and B channels each have their corresponding set of target pixels, and that a specific type of pixel only includes non-defective pixels. In this case, for the R channel, all non-defective pixels (i.e., those whose defect localization results in the R channel indicate they belong to the non-defective class) can be selected from its corresponding set of target pixels, and the defective channel components of these non-defective pixels in the R channel can be averaged to obtain the average defective channel component in the R channel. Similarly, for the B channel, all non-defective pixels (i.e., those whose defect localization results in the B channel indicate they belong to the non-defective class) can be selected from its corresponding set of target pixels, and the defective channel components of these non-defective pixels in the B channel can be averaged to obtain the average defective channel component in the B channel. Finally, for the G channel, the channel component of the i-th composite defective pixel in the G channel can be directly used as the channel component of the pixel mean in the G channel. Thus, the channel components of the pixel mean in the R, G, and B channels were obtained, which means the complete pixel mean was obtained.

[0096] Optionally, when averaging the defective channel components of a specific type of pixels in a set of target pixels, a weighted average method can be used, and the weights corresponding to all types of pixels in the specific type of pixels can be the same. Alternatively, when averaging the defective channel components of a specific type of pixels in a set of target pixels, a weighted average method can be used, and the weights corresponding to any two different types of pixels in the specific type of pixels can be different. For example, if the specific type of pixels includes both channel defective pixels and channel non-defective pixels, the weight corresponding to the channel defective pixels can be less than the weight corresponding to the channel non-defective pixels.

[0097] In one example, after calculating the pixel mean of the i-th composite defective pixel, this pixel mean can be directly used as the new pixel value for the i-th composite defective pixel. In another example, after calculating the pixel mean of the i-th composite defective pixel, its pixel mean and original pixel value can be blended to obtain the new pixel value for the i-th composite defective pixel. When blending the pixel mean and original pixel value of the i-th composite defective pixel, the blending can be performed directly based on the color space to which the pixel mean and original pixel value belong, or the two can be converted to another color space before blending. The following describes the blending method for converting the two to a new color space.

[0098] The new color space can be a color space that includes a luminance channel. It is understood that in this color space, channels other than the luminance channel are non-luminance channels. Exemplarily, and not limitingly, a color space that includes a luminance channel can be such as YUV, HSV, HSI, LAB, etc. In the YUV space, the Y (Luminance, Luma) channel is the luminance channel; in the HSV space, the V (Value) channel is the luminance channel; in the HSI space, the I (Intensity) channel is the luminance channel; and in the LAB space, the L (Lightness) channel is the luminance channel. Those skilled in the art will understand that in the field of image processing, the physical meanings of information such as luminance and brightness of an image or pixel are very close and can be used interchangeably. This document uniformly refers to this information used to represent the brightness of an image or pixel as "luminance".

[0099] The following example illustrates the conversion between RGB and YUV image space. Assuming the original image to be processed is an RGB image, each pixel in it has R, G, and B channel components. Correspondingly, the pixel mean and original pixel value corresponding to any composite defect pixel (still denoted as the i-th composite defect pixel) also have R, G, and B channel components. The pixel mean a1 and original pixel value s1 corresponding to the i-th composite defect pixel can be converted from RGB space to YUV space to obtain the converted pixel mean a2 and converted pixel value s2.

[0100] For the luminance channel component a in the transformed pixel mean a2 2I and the luminance channel component s in the transformed pixel value s2 2IThe first weighting relationship is used to perform a weighted average on the two. For the remaining non-luminance channel components in the transformed pixel mean a2 and the remaining non-luminance channel components in the transformed pixel value s2, a second weighting relationship can be used for weighted averaging. Of course, for any two different non-luminance channel components, they can correspond to the same second weighting relationship or different second weighting relationships.

[0101] The first and second weighting relationships can be set and adjusted independently. However, regardless of the relationship, the weight corresponding to the luminance channel component in the transformed pixel mean in the first weighting relationship is not 0, meaning the luminance channel component in the transformed pixel mean will always play a role in the new pixel value. For any non-luminance channel, the corresponding non-luminance channel component in the transformed pixel value can be directly used as the corresponding non-luminance channel component in the new pixel value, or the corresponding non-luminance channel component in the transformed pixel value can be mixed with the corresponding non-luminance channel component in the transformed pixel mean to obtain the corresponding non-luminance channel component in the new pixel value.

[0102] Because the human eye is sensitive to brightness in an image, the luminance channel component can be separated and processed separately from other non-luminance channel components. This allows for individual suppression of brightness. If the luminance channel component is weighted the same as other non-luminance channel components, and this weighting results in a particularly drastic change in the overall brightness of defective pixels, it may lead to "overcorrection," causing a particularly dark defect to become a particularly bright one. Therefore, processing the luminance channel component separately from other non-luminance channels allows for setting a more appropriate weighting relationship specifically for brightness, thus contributing to a more suitable and user-friendly defect removal effect.

[0103] According to an embodiment of this application, identifying defective regions from an image to be processed includes: performing image enhancement processing on the image to be processed to obtain an enhanced image; performing binarization processing on the enhanced image or a grayscale image based on a target pixel value threshold to obtain a mask image, wherein the grayscale image is obtained by converting the enhanced image, and in the mask image, pixels with a first pixel value belong to the defect category, and pixels with a second pixel value belong to the non-defect category; and determining the location of the defective region in the image to be identified based on the mask image.

[0104] Image enhancement processing can be performed on the image to be processed. This enhancement can be understood as image enhancement targeting defects, with the main purpose of strengthening the features of defects in the image to make them more prominent and identifiable compared to other image information. Image enhancement processing can be performed using any existing or future suitable image enhancement methods. For example, and not as a limitation, the CLAHE algorithm or high-frequency separation algorithms can be used for image enhancement processing. The CLAHE algorithm and high-frequency separation algorithms are efficient and fast; therefore, using these algorithms to identify defect regions can improve the efficiency and real-time performance of defect identification.

[0105] The CLAHE algorithm is as follows:

[0106] Step 1: Divide the image to be processed into m*n rectangular regions, where each rectangular region contains the same number of pixels. Then, calculate the histogram, cumulative distribution function, and transformation function for each rectangular region. The cumulative distribution function is obtained based on the histogram, and the transformation function is obtained based on the cumulative distribution function.

[0107] Step 2: Adjust the transformation function using the contrast limit value cl.

[0108] Step 3: Divide the pixels in the image to be processed into three cases according to their positions: pixels in the corner region are mapped according to the transformation function of the sub-image in which they are located; pixels in the edge region are mapped according to the transformation function of the two adjacent sub-images in which they are located and then linear interpolation is performed; pixels in the center region are mapped according to the transformation function of the four adjacent sub-images in which they are located and then bilinear interpolation is performed.

[0109] After the above three steps, the desired enhanced image A can be obtained.

[0110] The high-frequency separation algorithm is as follows:

[0111] Step 1: Blur the image to be processed to generate a low-frequency image.

[0112] Any suitable low-pass filter can be used to blur the image to be processed. Different filtering parameters will produce different results, so appropriate filtering parameters can be selected as needed to minimize defects in the low-frequency image obtained after filtering.

[0113] Step 2: Subtract the low-frequency image from the image to be processed to generate a high-frequency image (i.e., enhanced image) A. In the high-frequency image, positive values ​​represent brighter blemishes, and negative values ​​represent darker blemishes.

[0114] After obtaining the enhanced image A, a target pixel value threshold t can be used to binarize the enhanced image A or the grayscale image obtained by transforming the enhanced image A, resulting in a mask image M used to indicate the location of defective regions. For example, the mask image can be a black and white image, where white portions represent pixels with a first pixel value (representing defects) and black portions represent pixels with a second pixel value (representing non-defects). The first and second values ​​can be arbitrary values; for example, one can be 1 and the other 0.

[0115] The color space corresponding to the enhanced image A can contain multiple color channels, such as R, G, and B channels. In one example, the three color channels can be binarized separately to obtain three mask channel images (for easy distinction from the overall mask image, the mask image corresponding to each channel is called the mask channel image). In another example, the enhanced image A can be converted into a grayscale image, leaving only one channel. This single channel can then be binarized using a single threshold to obtain a mask image M.

[0116] After obtaining the mask image, for each pixel on the original image to be processed, it can be determined whether the pixel belongs to the defect category based on the pixel value at the corresponding position of the mask image M, thereby determining the location of the defect area.

[0117] When multiple mask channel images are obtained for each color channel, a defect location result can be obtained based on each mask channel image. Therefore, multiple defect location results corresponding one-to-one with multiple color channels can be obtained. Conversely, when only a single mask image is obtained, the same defect location result corresponding to all color channels can be obtained.

[0118] Image enhancement and binarization can be used to quickly and easily determine the location of defects.

[0119] According to embodiments of this application, the image to be processed includes one or more image channels corresponding one or more color channels; the enhanced image includes one or more enhanced image channels corresponding one or more color channels; the mask image includes one or more mask image channels corresponding one or more color channels; and the target pixel value threshold includes pixel value thresholds corresponding one or more color channels.

[0120] Binarizing the enhanced image or grayscale image based on a target pixel value threshold to obtain a mask image includes:

[0121] For each of one or more color channels, the enhancement channel image corresponding to that color channel is binarized based on the pixel value threshold corresponding to that color channel to obtain the mask channel image corresponding to that color channel.

[0122] Determining the location of defective regions in an image to be identified based on a mask image includes:

[0123] For each of one or more color channels, based on the mask channel image corresponding to that color channel, determine the location of the defect in the channel image to be processed corresponding to that color channel, so as to obtain the defect location result corresponding to that color channel;

[0124] The location of the defect area is determined based on one or more defect location results that correspond one-to-one with one or more color channels.

[0125] When there are multiple color channels, each color channel can have its own pixel value threshold. The pixel value thresholds corresponding to any two different color channels can be the same or different. In any case, since the image has a large number of pixels and each color channel has its own component values, the mask channel images corresponding to different color channels obtained through binarization are likely to be not completely identical.

[0126] As described above, when multiple mask channel images are obtained for each color channel, a defect localization result can be obtained based on each mask channel image. Therefore, multiple defect localization results corresponding one-to-one with multiple color channels can be obtained. Regardless of whether one or multiple defect localization results are obtained, the final defect region location can be determined by combining all obtained defect localization results. As described above, for any pixel on the image to be processed, if there are a target number of defect localization results indicating that the pixel belongs to the defect category, then the pixel can be determined as a comprehensive defect pixel, and the defect region includes at least a portion of the comprehensive defect pixels in the image to be processed.

[0127] By processing each color channel independently to obtain the corresponding mask channel image, the corresponding blemish location result can be obtained. This approach has higher accuracy in blemish location and helps to improve the blemish removal effect.

[0128] According to embodiments of this application, before binarizing the enhanced image or grayscale image based on a target pixel value threshold to obtain a mask image, identifying defective regions from the image to be processed further includes: converting the enhanced image into a grayscale image; binarizing the enhanced image or grayscale image based on a target pixel value threshold to obtain a mask image includes: binarizing the grayscale image based on a target pixel value threshold to obtain a mask image.

[0129] Those skilled in the art will understand how any image can be converted to a grayscale image, and this will not be elaborated upon here. By converting the enhanced image to a grayscale image, a single mask image can be obtained, that is, a single defect localization result can be obtained. This approach requires less computation and helps to improve the efficiency of image processing.

[0130] According to an embodiment of this application, the method further includes: identifying a specific region from the image to be processed, wherein the specific region includes one or more of the following types: hair region, clothing region, eye region, mouth region, eyebrow region, ear region, and region other than the face; and setting pixels within the specific region to belong to non-defective regions.

[0131] It's important to note that areas other than the face are a different area type from hair and clothing areas. Areas other than the face refer to all areas except the face itself, and their scope is larger compared to hair and clothing areas. Some users may want to exclude all areas outside the face from blemish removal; in this case, they can choose the "Areas other than the face" area type. Other users may only want to exclude hair and clothing areas from blemish removal; in this case, they can choose either the "Hair Area" or "Clothing Area" area types.

[0132] In the process of identifying defective areas, pixels in certain specific areas can be assumed to be non-defective by default. For example, this can be achieved by setting the pixel value of the pixel corresponding to the specific area in the mask image to a second value. That is, pixels in these specific areas can be disregarded, and even if these pixels were previously identified as defective during the calculation of the mask image, they can be further corrected to be non-defective later.

[0133] The aforementioned specific areas can be hair, clothing, eyes, mouth, eyebrows, ears, or areas other than the face. Generally, hair, clothing, eyes, etc., are considered flawless by default. Flaws identified in these areas are more likely to be false positives, or even if they are flaws, they may not significantly affect the aesthetics. Therefore, pixels in these areas can be ignored, and pixel replacement operations can be avoided. This helps improve the accuracy of flaw removal while reducing the system's workload, thus increasing the efficiency of flaw removal.

[0134] According to an embodiment of this application, before identifying a specific region from an image to be processed, the method further includes: determining the region type included in the specific region in response to type indication information related to the specific region input by a user.

[0135] Users can input type indication information into the system (e.g., the electronic device 100) through an input device (e.g., the input device 106 described above). The system determines the region type selected by the user based on the type indication information. For example, assuming the user indicates that the hair region and clothing region are specific region types, then during the identification of defective regions, all pixels contained in the hair region and clothing region in the image to be processed can be directly regarded as non-defective. This can be achieved by setting pixel values ​​in the mask image as described above.

[0136] The above scheme allows users to choose the specific area type they want to exclude (as a blemish) according to their needs, making the identification of blemish areas more flexible and improving the user experience. For example, a user who wants to remove blemishes from their entire face can select areas outside the face as the specific area. Or, a user who wants to remove blemishes from their cheeks, forehead, etc., but not from their eyes and mouth, can choose to select areas outside the face, as well as the eye and mouth areas, as the specific areas.

[0137] Of course, optionally, the type of area included in a specific area can also be set by default by the system (e.g., electronic device 100) as an inherent type or randomly selected by the system before de-defecting.

[0138] According to embodiments of this application, pixel replacement operations corresponding to all or part of the composite defective pixels in at least a portion of the composite defective pixels are executed in parallel with each other.

[0139] Because this application performs independent and consistent pixel replacement operations for each composite defect pixel, it is very convenient to execute pixel replacement operations for different composite defect pixels in parallel. Optionally, any existing or future processing device capable of parallel pixel operations, such as a graphics processing unit (GPU), can be used to execute pixel replacement operations for different composite defect pixels in parallel.

[0140] According to another aspect of this application, an image processing apparatus is provided. Figure 4 A schematic block diagram of an image processing apparatus 400 according to one embodiment of this application is shown.

[0141] like Figure 4 As shown, the image processing apparatus 400 according to an embodiment of this application includes an acquisition module 410, an identification module 420, and a pixel replacement module 430, wherein the pixel replacement module 430 includes a search submodule 432, a calculation submodule 434, and a replacement submodule 436. Each module can respectively perform the functions described above in conjunction with... Figures 2a-2bThe image processing method described herein comprises various steps / functions. The following description focuses only on the main functions of each component of the image processing apparatus 400, omitting the details already described above.

[0142] The acquisition module 410 is used to acquire the image to be processed, which includes the human skin region. The acquisition module 410 can be... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0143] The recognition module 420 is used to identify defect regions from the image to be processed. The initial color space corresponding to the image to be processed has one or more color channels, each corresponding to the same defect localization result, or each color channel corresponds one-to-one with one or more defect localization results. For any pixel in the image to be processed, if a target number of defect localization results indicate that the pixel belongs to a defect class, then the pixel is a composite defect pixel. The defect region contains at least a portion of the composite defect pixels in the image to be processed. The target number is greater than or equal to 1 and less than or equal to the total number of one or more defect localization results. The recognition module 420 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0144] The pixel replacement module 430 is used to perform a pixel replacement operation for each composite defect pixel in at least a portion of the composite defect pixels in the defect region. The pixel replacement module 430 can be... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0145] The lookup submodule 432 is used to, for each defect location result indicating that the overall defect pixel belongs to a defect class, find a set of target pixels surrounding the overall defect pixel that meet the target requirements, wherein the set of target pixels meeting the target requirements includes at least one channel non-defect pixel, and the channel non-defect pixel is the pixel that the defect location result indicates belongs to a non-defect class. The lookup submodule 432 can be... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0146] The calculation submodule 434 is used to calculate a new pixel value for the composite defective pixel, based at least on the pixel value of the found target pixel. The calculation submodule 434 can be... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0147] The replacement submodule 436 is used to replace the original pixel value of the composite defective pixel in the image to be processed with the new pixel value of the composite defective pixel. The replacement submodule 436 can be... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0148] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0149] Figure 5 A schematic block diagram of an electronic device 500 according to an embodiment of this application is shown. The electronic device 500 includes a storage device (i.e., a memory) 510 and a processor 520.

[0150] Storage device 510 stores computer program instructions for implementing corresponding steps in the image processing method 200 according to an embodiment of the present application.

[0151] The processor 520 is used to run computer program instructions stored in the storage device 510 to perform corresponding steps of the image processing method 200 according to an embodiment of the present application.

[0152] In one embodiment, computer program instructions executed by processor 520 are used to perform the following steps: acquiring an image to be processed, the image to be processed including a human skin region; identifying blemish regions from the image to be processed, wherein the initial color space corresponding to the image to be processed has one or more color channels, the one or more color channels corresponding to the same blemish localization result, or the one or more color channels corresponding one-to-one with one or more blemish localization results, wherein for any pixel on the image to be processed, if there is a target number of blemish localization results indicating that the pixel belongs to a blemish class, then the pixel is a composite blemish pixel, the blemish region contains at least a portion of the composite blemish pixels in the image to be processed, and the target number is greater than or equal to 1 and less than or equal to The total number of one or more defect localization results; for each composite defect pixel in at least a portion of composite defect pixels in the defect region, perform the following pixel replacement operation: for each defect localization result indicating that the composite defect pixel belongs to a defect class, based on the defect localization result, find a set of target pixels around the composite defect pixel that meet the target requirements, wherein the set of target pixels that meet the target requirements includes at least one channel non-defect pixel, the channel non-defect pixel being a pixel that the defect localization result indicates belongs to a non-defect class; calculate a new pixel value for the composite defect pixel based at least on the pixel value of the found target pixels; replace the original pixel value of the composite defect pixel on the image to be processed with the new pixel value of the composite defect pixel.

[0153] Exemplarily, the electronic device 500 may also include an image acquisition device 530. The image acquisition device 530 is used to acquire a raw image or an image to be processed, the image to be processed being obtained based on the raw image. The image acquisition device 530 is optional; the electronic device 500 may not include it. In this case, other image acquisition devices can be used to acquire the raw image or the image to be processed, and the acquired image to be processed can be sent to the electronic device 500.

[0154] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When the program instructions are run by a computer or processor, they are used to execute corresponding steps of the image processing method of the embodiments of this application and to implement corresponding modules in the image processing apparatus according to the embodiments of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0155] In one embodiment, when the program instructions are executed by a computer or processor, the computer or processor may implement the various functional modules of the image processing apparatus according to the embodiments of the present application, and / or may execute the image processing method according to the embodiments of the present application.

[0156] In one embodiment, the program instructions are used at runtime to perform the following steps: acquiring an image to be processed, the image to be processed including a human skin region; identifying blemish regions from the image to be processed, wherein the initial color space corresponding to the image to be processed has one or more color channels, the one or more color channels corresponding to the same blemish localization result, or the one or more color channels corresponding one-to-one with one or more blemish localization results, wherein for any pixel on the image to be processed, if there is a target number of blemish localization results indicating that the pixel belongs to a blemish class, then the pixel is a composite blemish pixel, the blemish region contains at least a portion of the composite blemish pixels in the image to be processed, and the target number is greater than or equal to 1 and less than or equal to one or more. The total number of defect localization results; for each composite defect pixel in at least a portion of composite defect pixels in the defect region, perform the following pixel replacement operation: for each defect localization result indicating that the composite defect pixel belongs to a defect class, based on the defect localization result, find a set of target pixels around the composite defect pixel that meet the target requirements, wherein the set of target pixels that meet the target requirements includes at least one channel non-defect pixel, the channel non-defect pixel being a pixel that the defect localization result indicates belongs to a non-defect class; calculate a new pixel value for the composite defect pixel based at least on the pixel value of the found target pixels; replace the original pixel value of the composite defect pixel on the image to be processed with the new pixel value of the composite defect pixel.

[0157] Furthermore, according to an embodiment of this application, a computer program product is also provided, the computer program product including a computer program, which, when running, is used to execute the above-described image processing method 200.

[0158] Each module in the electronic device according to the embodiments of this application can be implemented by the processor of the electronic device implementing image processing according to the embodiments of this application running computer program instructions stored in memory, or by computer instructions stored in a computer-readable storage medium of a computer program product according to the embodiments of this application being executed by a computer.

[0159] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0160] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0162] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0163] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more aspects of the various applications, features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, the inventive point lies in solving the corresponding technical problem with fewer features than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0164] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0165] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0166] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the image processing apparatus according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0167] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0168] The above are merely specific embodiments or descriptions of specific embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. An image processing method, comprising: Acquire an image to be processed, the image to be processed including a human skin region; Identify defective regions from the image to be processed, wherein the initial color space corresponding to the image to be processed has one or more color channels, the one or more color channels correspond to the same defect location result, or the one or more color channels correspond one-to-one with one or more defect location results, wherein for any pixel on the image to be processed, if there is a target number of defect location results indicating that the pixel belongs to the defect category, then the pixel is a composite defect pixel, the defective region contains at least a portion of the composite defect pixels in the image to be processed, and the target number is greater than or equal to 1 and less than or equal to the total number of the one or more defect location results; For each composite defect pixel in at least a portion of the composite defect pixels in the defective region, perform the following pixel replacement operation: For each defect location result indicating that the overall defect pixel belongs to the defect category, based on the defect location result, a set of target pixels that meet the target requirements are found around the overall defect pixel. The set of target pixels that meet the target requirements includes at least one channel non-defect pixel, which is a pixel that the defect location result indicates belongs to the non-defect category. Based at least on the pixel values ​​of the found target pixels, a new pixel value for the composite defective pixel is calculated, including: for each defective channel of the composite defective pixel, averaging the defective channel components of at least a portion of the target pixels in a group of target pixels corresponding to that defective channel to obtain a pixel mean, wherein at least a portion of the target pixels corresponding to any defective channel include only channel-non-defective pixels corresponding to that defective channel or include both channel-non-defective pixels and channel-defective pixels corresponding to that defective channel, and the channel-defective pixels corresponding to any defective channel are pixels whose defect localization results indicate that they belong to the defect category; the pixel mean and The original pixel values ​​of the composite defective pixels are converted to a new color space that includes a luminance channel to obtain a converted pixel mean and a converted pixel value. The luminance channel is used to represent the brightness of a pixel. The new pixel value is obtained based on the luminance channel component and the non-luminance channel component of the new pixel value. The luminance channel component of the new pixel value is determined based on the luminance channel component in the converted pixel mean and the luminance channel component in the converted pixel value. The non-luminance channel component of the new pixel value is determined based on the non-luminance channel component in the converted pixel mean and the non-luminance channel component in the converted pixel value. The original pixel value of the composite defect pixel in the image to be processed is replaced with the new pixel value of the composite defect pixel.

2. The method as described in claim 1, wherein, For each defect localization result indicating that the overall defect pixel belongs to a defect category, the process of finding a set of target pixels that meet the target requirements around the overall defect pixel based on the defect localization result includes: For each defect localization result indicating that the overall defect pixel belongs to the defect class, perform the following lookup operation: Step a: Based on the defect location result, determine whether the pixels in the current search range meet the target requirements. If they do, proceed to step b; otherwise, proceed to step c. Step b: Determine the pixels within the current search range as a set of target pixels that meet the target requirements and end the search; Step c: Determine a new range around the composite defective pixel based on the current search range, wherein the inner boundary of the new range coincides with the outer boundary of the current search range, or the inner boundary of the new range is further out than the outer boundary of the current search range; Step d: Determine the new range as the current search range and return to step a.

3. The method as described in claim 2, wherein, The current search range is the area covered by pixels that are a first chessboard distance away from the defective pixel, and the new range is the area covered by pixels that are a second chessboard distance away from the defective pixel. The distance of the second chessboard is greater than the distance of the first chessboard, and the distance of the second chessboard differs from the distance of the first chessboard by one or more chessboard squares, with each chessboard square being one pixel.

4. The method as described in any one of claims 1 to 3, wherein, The proportion of channel-free pixels in a set of target pixels that meet the target requirements is greater than or equal to the target proportion threshold, and / or the number of channel-free pixels in a set of target pixels that meet the target requirements is greater than or equal to the target number threshold.

5. The method as described in any one of claims 1 to 3, wherein, The found target pixel is the union of one or more sets of target pixels that correspond one-to-one with one or more defect channels of the overall defect pixel. The defect channel is the color channel that indicates the defect category of the overall defect pixel according to the defect localization result. Before obtaining the new pixel value based on the luminance channel component and the non-luminance channel component of the new pixel value, the calculation of the new pixel value of the composite defective pixel, at least based on the pixel value of the found target pixel, further includes: The luminance channel component in the mean value of the converted pixel and the luminance channel component in the converted pixel value are weighted and averaged using a first weighting relationship to obtain the luminance channel component of the new pixel value. The non-brightness channel component in the mean value of the converted pixel and the non-brightness channel component in the converted pixel value are weighted and averaged using a second weighting relationship to obtain the non-brightness channel component of the new pixel value. In the first weighted relationship, the weight corresponding to the luminance channel component in the converted pixel mean is greater than 0, and in the second weighted relationship, the weight corresponding to the non-luminance channel component in the converted pixel mean is equal to or greater than 0.

6. The method as described in any one of claims 1 to 3, wherein, The step of identifying defective regions from the image to be processed includes: The image to be processed is subjected to image enhancement processing to obtain an enhanced image; The enhanced image or grayscale image is binarized based on a target pixel value threshold to obtain a mask image. The grayscale image is obtained by converting the enhanced image. In the mask image, pixels with a first pixel value belong to the defect category, and pixels with a second pixel value belong to the non-defect category. The location of the defective region in the image to be processed is determined based on the mask image.

7. The method of claim 6, wherein, The image to be processed includes one or more image channels corresponding to the one or more color channels; the enhanced image includes one or more image channels corresponding to the one or more color channels; the mask image includes one or more image channels corresponding to the one or more color channels; and the target pixel value threshold includes pixel value thresholds corresponding to the plurality of color channels. The step of binarizing the enhanced image or grayscale image based on a target pixel value threshold to obtain a mask image includes: For each of the one or more color channels, the enhancement channel image corresponding to the color channel is binarized based on the pixel value threshold corresponding to the color channel to obtain the mask channel image corresponding to the color channel. Determining the location of the defective region in the image to be processed based on the mask image includes: For each of the one or more color channels, based on the mask channel image corresponding to that color channel, the location of the defect in the channel image to be processed corresponding to that color channel is determined, so as to obtain the defect location result corresponding to that color channel; The location of the defect area is determined based on one or more defect location results that correspond one-to-one with the one or more color channels.

8. The method as described in any one of claims 1 to 3, wherein, The method further includes: Identify specific regions from the image to be processed, wherein the specific regions include one or more of the following types: hair region, clothing region, eye region, mouth region, eyebrow region, ear region, and region other than the face; Pixels within the specified area are set to belong to the non-defective area.

9. The method as described in any one of claims 1 to 3, wherein, The pixel replacement operations corresponding to all or part of the composite defective pixels in the at least some composite defective pixels are performed in parallel with each other.

10. An electronic device comprising a processor and a memory, wherein, The memory stores computer program instructions, which, when executed by the processor, are used to perform the image processing method as described in any one of claims 1 to 9.

11. A storage medium storing program instructions that, when executed, perform the image processing method as claimed in any one of claims 1 to 9.

12. A computer program product, the computer program product comprising a computer program, characterized in that, The computer program, when running, is used to perform the image processing method as described in any one of claims 1 to 9.

Images