Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method, an electronic device and a storage medium, relates to the technical field of terminals, and can solve the problem that satisfactory shooting images are sometimes difficult to obtain due to reflection. The image processing method provided by the application comprises the following steps: acquiring a source image collected by a camera device; identifying a reflection pattern in the source image, and generating a reflection level representing the strength of the reflection pattern according to the identified reflection pattern, wherein the reflection pattern is an image formed in the source image due to reflection; and when the reflection level is less than or equal to a first preset value, saving the source image in a memory. The image processing method can be applied to terminal products.

Description

Technical Field

[0001] This application relates to the field of terminal technology, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] When photographing an object with reflective properties, the luminous object behind the camera leaves its image in the captured photograph. These images appear as reflection patterns on the surface of the reflective object. These reflection patterns are difficult to eliminate and often affect image quality. Summary of the Invention

[0003] In view of this, this application provides an image processing method, apparatus, electronic device, and storage medium that can solve the problem of not being able to obtain satisfactory captured images due to reflection phenomena.

[0004] Embodiments of this application provide an image processing method, including:

[0005] Acquire source images captured by the camera device;

[0006] Identify reflection patterns in a source image and generate a reflection level characterizing the strength of the reflection pattern based on the identified reflection pattern, wherein the reflection pattern is an image formed in the source image due to reflection phenomena;

[0007] When the reflection level is less than or equal to a first preset value, the source image is saved in the memory.

[0008] Embodiments of this application also provide an image processing apparatus, comprising:

[0009] The acquisition unit is used to acquire source images;

[0010] The recognition unit is used to recognize the reflection pattern in the source image;

[0011] A reflection level acquisition unit is used to identify reflection patterns in a source image and generate a reflection level characterizing the strength of the reflection pattern based on the identified reflection pattern, wherein the reflection pattern is an image formed in the source image due to reflection phenomena.

[0012] A storage unit is used to store the source image when the reflection level is less than or equal to a first preset value.

[0013] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a program for being executed by the processor to perform the image processing method described above.

[0014] Embodiments of this application provide a readable storage medium storing a program for execution by a processor to perform the image processing method as described.

[0015] This application provides an image processing method, electronic device, and storage medium. After acquiring a source image from a camera device, before saving the source image to memory, a reflection pattern in the source image is first identified, and a reflection level characterizing the strength of the reflection pattern is generated based on the identified reflection pattern. The reflection pattern is the image formed in the source image due to reflection. When the reflection level is less than or equal to a first preset value, i.e., the reflection pattern is weak, the source image is saved to memory. This solves the problem of not obtaining a satisfactory captured image due to reflection. This solution saves storage space by only saving the source image to memory when the reflection pattern in the source image is weak. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1(a) is a photograph of a bookshelf taken directly. Figure 1(b)-Figure 1(d) The photo shows the same bookshelf taken through glass. Figure 1(b)-Figure 1(d) The reflection pattern gradually intensifies;

[0018] Figure 2 A hardware schematic diagram of an electronic device for implementing the various embodiments of this application;

[0019] Figure 3 This is a schematic flowchart of an image processing method according to an embodiment of this application;

[0020] Figure 4 This is a schematic flowchart illustrating the generation of reflection levels in an image processing method according to an embodiment of this application.

[0021] Figure 5 This is a schematic flowchart illustrating another method for generating reflection levels in an image processing procedure according to an embodiment of this application.

[0022] Figure 6 This is a schematic flowchart of an image processing method for edge detection of a depth image according to an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of a process for obtaining the reflection level according to an embodiment of this application;

[0024] Figure 8 A schematic diagram illustrating the process of an electronic device taking photos according to an embodiment of this application;

[0025] Figure 9 A schematic diagram illustrating the process of an electronic device taking photographs according to another embodiment of this application;

[0026] Figure 10 This application provides a hardware schematic diagram of an image processing device;

[0027] Figure 11 A hardware schematic diagram of the reflection level acquisition unit of the image processing apparatus provided in this application;

[0028] Figure 12 This application provides a hardware schematic diagram of an electronic device. Detailed Implementation

[0029] Figure 1(a) is a photograph of a bookcase taken directly. Figure 1(b)-Figure 1(d) These are photographs taken through glass of the same bookcase. The window behind the camera leaves varying degrees of reflection patterns in the images. These different reflection patterns have varying effects on the captured images. Compared to the original image Figure 1(a), the reflection pattern in Figure 1(b) is weaker, and its impact on the captured image is negligible; Figure 1(b) is also a satisfactory photograph. Conversely, the reflection pattern in Figure 1(d) is very strong, and its impact on the captured image is not negligible; Figure 1(d) is generally not satisfactory to users. The reflection pattern in Figure 1(c) has an impact on the captured image that falls between Figure 1(b) and Figure 1(d). However, in actual shooting, it is difficult for users to judge the strength of the reflection pattern and its impact on the captured image with the naked eye from the camera monitor. Often, after shooting, it is found that the captured image is unsatisfactory due to the presence of these reflection patterns.

[0030] The purpose of this disclosure is to provide an image processing method that solves the aforementioned problems. Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] It should be noted that, in this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0032] It should be further understood that the terms "comprising" or "including" indicate the presence of the said features, steps, operations, elements, components, items, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, types, and / or groups.

[0033] It should be understood that although the terms first, second, third, etc., may be used in this document to describe various types of information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this document, a first edge image may also be referred to as a second edge image, and similarly, a second edge image may also be referred to as a first edge image.

[0034] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0035] It should be noted that step codes such as 201 and 202 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute 303 first and then 302, etc., but these should all be within the protection scope of this application.

[0036] In the following description, suffixes such as “module,” “part,” or “unit” used to denote elements are used only for the purposes of this application and have no specific meaning in themselves.

[0037] The electronic device described below can be any electronic device that requires image processing. Generally, the electronic device described above can be a mobile terminal. Mobile terminals can be implemented in various forms. For example, the mobile terminals described in this application can include mobile terminals such as mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0038] The following description will use a mobile terminal as an example. Those skilled in the art will understand that, apart from elements specifically designed for mobile purposes, the construction according to the embodiments of this application can also be applied to fixed-type terminals or other electronic devices.

[0039] Please see Figure 2 This is a hardware schematic diagram of an electronic device implementing various embodiments of this application. The electronic device 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (Audio / Video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111, etc. Those skilled in the art will understand that... Figure 2 The electronic device hardware shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0040] The following is combined Figure 2 A detailed introduction to each component of the electronic device:

[0041] The radio frequency unit 101 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 110; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, and a duplexer. Furthermore, the radio frequency unit 101 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), and TDD-LTE (Time Division Duplexing-Long Term Evolution).

[0042] WiFi is a short-range wireless transmission technology. Electronic devices using the WiFi module 102 can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 2 WiFi module 102 is shown, but it is understood that it is not a necessary component of an electronic device and can be omitted as needed without changing the nature of the invention.

[0043] The audio output unit 103 can convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into audio signals and output them as sound when the electronic device 100 is in call signal receiving mode, call mode, recording mode, voice recognition mode, broadcast receiving mode, etc. Furthermore, the audio output unit 103 can also provide audio output related to specific functions performed by the electronic device 100 (e.g., call signal receiving sound, message receiving sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0044] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on the display unit 106. The image frames processed by the GPU 1041 can be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) in operating modes such as telephone call mode, recording mode, and voice recognition mode, and can process such sound into audio data. The processed audio (voice) data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 101 in telephone call mode. The microphone 1042 can implement various types of noise cancellation (or suppression) algorithms to eliminate (or suppress) noise or interference generated during the reception and transmission of audio signals.

[0045] The electronic device 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the ambient light level, and the proximity sensor can turn off the display panel 1061 and / or backlight when the electronic device 100 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify device posture (such as switching between portrait and landscape modes on a mobile phone, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that can also be configured in a mobile phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0046] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0047] User input unit 107 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 107 may include touch panel 1071 and other input devices 1072. Touch panel 1071, also known as touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 1071), and drive corresponding connection devices according to a pre-set program. Touch panel 1071 may include two parts: a touch detection device and a touch controller. Optionally, the touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to processor 110, and can receive and execute commands sent by processor 110. In addition, touch panel 1071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc., without being limited here.

[0048] Furthermore, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Subsequently, the processor 110 provides corresponding visual output on the display panel 1061 based on the type of touch event. Although in Figure 2 In this embodiment, the touch panel 1071 and the display panel 1061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.

[0049] Interface unit 108 serves as an interface through which at least one external device can connect to electronic device 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 108 may be used to receive input (e.g., data, power, etc.) from the external device and transmit the received input to one or more components within electronic device 100, or it may be used to transmit data between electronic device 100 and the external device.

[0050] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a program storage area and a data storage area. Optionally, the program storage area may store the operating system, application programs required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, the memory 109 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0051] The processor 110 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 109, and by calling data stored in the memory 109, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor. Optionally, the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 110.

[0052] The electronic device 100 may also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 111 can be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0053] although Figure 2 As not shown, the electronic device 100 may also include a Bluetooth module, etc., which will not be described in detail here.

[0054] Based on the above-described hardware structure of the electronic device, various embodiments of this application are proposed.

[0055] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application. It should also be understood that the electronic device hardware structures described herein are merely illustrative of this application and are not intended to limit this application.

[0056] like Figure 3 As shown, an embodiment of this application provides an image processing method, including:

[0057] 201. Acquire the source image captured by the camera device;

[0058] 202. Identify the reflection pattern in the source image, and generate a reflection level characterizing the strength of the reflection pattern based on the identified reflection pattern;

[0059] In this embodiment, the source image is an image acquired from a camera device and to be saved to a memory. In some embodiments, the source image is a pattern captured through an object with reflective properties, such as an image captured through glass. In this step, image processing is used to obtain a reflection level characterizing the strength of the reflective pattern; the specific method is not limited.

[0060] In this embodiment, the reflection pattern is the image formed in the source image by an object outside the shooting range due to reflection when photographed through an object with reflective properties. For example, when photographing an object behind glass, the image of a luminous object located behind the camera in the source image is reflected on the glass.

[0061] The intensity of a reflection pattern refers to the size of the distribution area corresponding to the reflection pattern and the density of the reflection pattern in the source image. The density of the reflection pattern refers to the proportion of pixel values ​​in the source image that correspond to the reflection pattern. As shown in Figures 1(b)-(d), the intensity of the reflection pattern on the bookshelf increases sequentially. Generally, the intensity of the reflection pattern is relative to the image of the target in the source image.

[0062] The reflectance level is used to characterize the strength of the aforementioned reflective pattern and can reflect the degree of influence of the reflective pattern on image quality. For example, the reflectance level can be roughly divided into three levels: very faint, faint, and strong. Different processing methods are used in this disclosure for different reflectance levels.

[0063] 203. When the reflection level is less than or equal to a first preset value, the source image is stored in the memory. The first preset value corresponds to a reflection intensity where the reflection pattern is weak and its impact on the shooting effect is negligible. The first preset value can be obtained through experimentation or artificial intelligence learning.

[0064] The image processing method provided in this embodiment first obtains the reflection level, which represents the strength of the reflection pattern, before saving the source image in the memory. When the reflection level is less than or equal to a first preset value, i.e. the reflection pattern is weak, the source image is saved in the memory. This can avoid storing low-quality images with strong reflection patterns and save storage space.

[0065] In some embodiments, the source image includes a color image and a depth image captured by a camera device of the same scene; in step 201, edges in the color image and the depth image are detected, and the reflection level is generated based on the edge detection results. Here, an edge in the color image or the depth image is a point in the color image or the depth image where the image intensity value is discontinuously greater than a predetermined threshold.

[0066] Reflective patterns in color images are generally the reflections left by luminous objects behind the camera. When a depth sensor is used to capture the subject, it acquires depth information but not information about the luminous objects behind the camera. Therefore, the depth image acquired by the depth sensor does not contain information about the reflective patterns. Color images, on the other hand, contain information about both the subject and the luminous objects behind the camera that are reflected onto the subject. Therefore, by analyzing the differences between the information contained in the depth and color images, information about the intensity and distribution area of ​​the reflective patterns can be obtained, thus determining the reflectivity level of the reflective patterns.

[0067] Furthermore, in this embodiment, edge information of the color image and edge information of the depth image are obtained by detecting the edges in the color image and the depth image, and the reflection level of the reflection pattern is obtained based on the difference in edge information contained in the color image and the depth image (the edges in the depth image do not include the edges of the reflection pattern).

[0068] In some embodiments, edges in the color image and the depth image are detected respectively, and edge matching is performed on the edges detected in the color image and the edges detected in the depth image. A reflectance level is generated based on the edge matching results. Edge matching refers to detecting whether the pixels of an edge in the color image correspond to edges in the depth image; if so, the match is successful; otherwise, the match fails. Unmatched or failed-matched pixels contain edge information of the reflectance pattern, which can be used to obtain the reflectance level of the reflectance pattern. The process of obtaining the reflectance level will be further described below with specific examples.

[0069] For example, such as Figure 4 As shown, in some embodiments, identifying a reflection pattern in the source image and generating a reflection level characterizing the strength of the reflection pattern based on the identified reflection pattern includes:

[0070] 301. Perform field-of-view matching and pixel correspondence matching processing on the color image and the depth image so that the processed color image and the depth image have the same field of view size and have a one-to-one pixel correspondence.

[0071] Color and distance / depth images are captured simultaneously. These images are then subjected to field-of-view matching and pixel-to-pixel correspondence matching, a step that can be performed offline or online. The final result of this step is two photographs of the scene—an RGB color image and a depth image of the same size. In the color image, any pixel (x, y) contains RGB pixel information, while the same pixel (x, y) in the depth image contains the depth (or distance) information from the depth sensor of the subject to the camera.

[0072] The specific implementation methods for field matching and pixel-to-pixel correspondence matching are not limited, as long as the processed color image and the depth image have the same field size and there is a one-to-one correspondence between pixels.

[0073] 302. Perform edge detection on the color image and generate a first edge image based on the edge detection results;

[0074] This step performs edge detection on the color image to identify edges representing the contours of objects in the scene. The purpose of edge detection is to identify pixels in the color image whose intensity discontinuities exceed a preset threshold, or simply put, to identify pixels with abrupt changes in image intensity. Exemplary edge detection algorithms include: Canny edge detection, Deriche edge detection, Sobel edge detection, Prewitt edge detection, etc. This step extracts edge patterns to generate a first edge image.

[0075] 303. Perform edge detection on the depth image and generate a second edge image based on the edge detection results;

[0076] This step also performs edge detection on the depth image to extract its boundaries or contours. Similarly, this can be achieved by identifying points in the depth image where discontinuities in the image depth values ​​exceed a predetermined threshold, or simply, by identifying points where the image depth values ​​change abruptly. Exemplary edge detection algorithms include: Canny edge detection, Deriche edge detection, Sobel edge detection, Prewitt edge detection, etc. This step extracts the edge pattern from the depth image to generate a second edge image.

[0077] 304. Perform edge matching on the first edge image and the second edge image to filter out pixels that appear as edges in both the first edge image and the second edge image, and generate a common edge image based on the filtering results;

[0078] As described in step 301, for a point in the image corresponding to a pixel (e.g., (x, y)), its color information is stored in the first pixel of the color image, and its corresponding depth information is stored in the second pixel of the depth image. That is, the first pixel and the second pixel with the same position information in the field of view are corresponding, respectively representing the color information and depth information of a point in the field of view.

[0079] Edge matching in this step refers to obtaining the position information (e.g., (x, y)) of any colored pixel that represents an edge in the first edge image, and then obtaining the depth value of the corresponding depth pixel in the second edge image based on this position information. If the depth value of the corresponding depth pixel also represents an edge, then the pixel with the position information (e.g., (x, y)) is a common edge pixel, and the relevant information is recorded in the common edge image. If the common edge image is a binary image, for example, the corresponding pixel value of the position information (e.g., (x, y)) in the common edge image can be assigned a value of 1. If the depth value of the corresponding depth pixel does not represent an edge, then the pixel value of the position information (e.g., (x, y)) in the common edge image can be assigned a value of 0. Finally, other pixels in the common edge image that have not undergone the above assignment operation are assigned a value of 0 or maintain a preset value of 0. The edges of the reflection pattern have been removed from the edges represented by the common edge image in this step.

[0080] 305. Determine the reflection level based on the common edge image and the first edge image.

[0081] For example, the unmatched edges contain information about the reflection pattern, which can be obtained from a first edge image (representing all edges in the color image, including the edges of the subject and the reflection pattern) and a common edge image (edges with the reflection pattern removed). The reflection level can be calculated based on the reflection pattern information contained in the reflection edge image.

[0082] For example, this step can first filter out pixel information that appears as an edge in the first edge image but does not appear as an edge in the common edge image, and generate a reflective edge image based on the obtained pixel information. Then, the ratio of the number of pixels that are edges in the reflective edge image to the number of pixels that are edges in the first edge image is determined to obtain the reflectance level.

[0083] This embodiment performs edge recognition using color and depth images, matches the recognized edges, and filters out common edge pixels that appear as edges in both the first and second edge images. The reflection level of the reflection pattern is then calculated based on the common edge pixels and the edges in the first edge image. The calculation is simple, the response is fast, and it is suitable for predicting the impact of the reflection pattern on the camera effect during terminal shooting.

[0084] In some embodiments, such as Figure 5As shown, before performing field-of-view matching and pixel-to-pixel matching on the color image and the depth image to make the processed color image and the depth image have the same field of view size and have a one-to-one pixel correspondence; and before performing edge detection on the color image and generating a first edge image based on the edge detection result, the image processing method further includes: 306, performing illuminance normalization processing on the color image to make the illuminance of the color image uniform.

[0085] This step can correct for any non-uniform illumination using various illuminance normalization methods, such as image histogram equalization. The corrected color image can help improve the accuracy of reflectance calculations.

[0086] In some embodiments, such as Figure 5 As shown, the image processing method further includes: 307, filling holes in the depth image to optimize its quality. Using a hole-filled depth image to calculate reflectance levels can improve calculation accuracy, preventing holes in the depth image from affecting the calculation results, especially when calculating reflectance levels using the number of common edge pixels.

[0087] In some embodiments, such as Figure 6 As shown, the steps for performing edge detection on a depth image and generating a second edge image based on the edge detection results include:

[0088] 3031. Divide the depth image into depth regions based on pixel depth values ​​to generate a first image with multiple grayscale ranges;

[0089] In this step, after the depth image undergoes hole filling, different depth slices of the image are obtained through histogram analysis. This step generates an image with multiple grayscale intervals, the values ​​of which can be obtained by performing an arbitrary peak detection algorithm on the intensity histogram.

[0090] 3032. Perform multi-level thresholding processing on the first image to generate multiple binary images, wherein each binary image corresponds to a grayscale range of the first image;

[0091] Multi-level thresholding can be understood as generating multiple binary images corresponding to multiple grayscale intervals in a first image. Specifically, for each grayscale interval in the first image, two thresholds can be applied to generate a binary image: one greater than a given intensity and the other less than a given intensity. For example, let the two grayscale peak intensities in the image be 150 and 100. To generate a binary image with a pixel intensity of 100 (or between 100 and 150), two thresholds, 99 and 149, can be applied. Applying these two thresholds will set all pixels with an intensity equal to or less than 99 and all pixels with an intensity greater than 149 (up to 255) to 0. All other pixels will be set to an intensity value of 255.

[0092] 3033. Perform edge detection on each of the multiple binary images, and generate a second edge image based on the edge detection results.

[0093] This step also performs edge detection on each binary image to extract its boundaries or contours and generate a second edge image. For example, a simple boundary extraction method can be implemented using image morphology operations. If the previous step generated multiple binary images (F1, F2…Fn), specifically, the edge detection process in this step includes:

[0094] First, for a given binary image such as Fm, an eroded / reduced image such as image Gm is obtained using a 1x1 (or 3x3) structuring element.

[0095] Next, image Hm is obtained by subtracting image Fm pixel by pixel from image Gm. This process is repeated for each binary image.

[0096] Finally, a bitwise OR operation is performed on all output images (G1, G2…Gn) to form an image with all contours, namely the second edge image.

[0097] When calculating the reflectance level, a bitwise subtraction operation can be performed on the common edge image and the first edge image to generate a reflectance edge image. This reflectance edge image represents the edges caused by reflection. The bitwise subtraction operation involves removing pixels from the common edge image that correspond to edges in the first edge image. After generating the reflectance edge image, the number of pixels representing edges in the reflectance edge image and the number of pixels representing edges in the first edge image are determined, and their ratio is calculated to obtain the reflectance level.

[0098] This embodiment first converts the depth image into multiple binary images, and then uses image morphology operations to implement a simple boundary extraction method to obtain the edges of the depth image, reducing the amount of computation and thus improving the response speed. It is suitable for predicting the impact of reflection patterns on the camera effect when the terminal is shooting.

[0099] The following is based on Figure 2 The technical solution of this application is illustrated using the electronic device shown as an example and in conjunction with specific embodiments. Those skilled in the art will understand that the electronic device described in this application is merely an example, and any other suitable electronic device is included within the scope of this application. The electronic device in this embodiment includes a color camera and a depth camera.

[0100] Figure 8 and Figure 9 A flowchart illustrating the process of an electronic device taking a photograph, as provided in this application. Figure 7 A flowchart for obtaining the reflection level provided in this application is given.

[0101] like Figure 8 As shown, the process of the electronic device provided in this application taking photos includes:

[0102] 801. Capture color and depth images.

[0103] This step captures color and depth images of the scene using a camera sensor and a distance sensor.

[0104] 802. Save the color image and depth image to the cache, i.e., the buffer memory. The cache in this step generally refers to the high-speed cache memory.

[0105] The image processing method provided in this embodiment first stores the source image in an erasable, memoryless buffer. Subsequent steps perform image processing to obtain the reflection level, which characterizes the strength of the reflection pattern, and determine whether to transfer the image to a non-buffered memory based on the strength of the reflection pattern. This provides users with an option to consider image quality before shooting, thereby achieving good photographic results and saving storage space.

[0106] 803. Perform image processing on color and depth images to obtain reflectance levels.

[0107] The steps for obtaining the reflection level can be based on the method for generating the reflection level provided in any of the above embodiments. For example, it can be achieved by... Figure 7 The illustrated process for obtaining reflectance levels includes: performing field-of-view matching and pixel matching on the color image and depth image to obtain a color image and a depth image with the same field of view size and a one-to-one pixel correspondence. The matched color image is then normalized for illumination correction, followed by edge detection. The matched depth image is then filled with holes, segmented based on depth, and subjected to multi-level thresholding to convert it into multiple binary images, followed by edge detection. Edge matching is performed on the edges of the color image and the depth image, and reflectance levels are generated based on the edge matching results.

[0108] 804. Determine whether the reflection level is greater than or equal to the first preset value.

[0109] 805. If the reflection level is greater than or equal to the first preset value, output the reflection level; for example, display the reflection level on the monitor.

[0110] 806. Receive the user's instruction to store or discard storage based on the reflection level. This step identifies whether the user instructs to store or discard storage.

[0111] 807. If the reflection level is less than or equal to the first preset value, or if the reflection level is greater than the first preset value but the user inputs a storage command, then the image is saved in the non-buffered memory.

[0112] 808. If the user inputs an instruction to discard the stored data, the color image and depth image will be deleted from the buffer memory.

[0113] Optionally, the above shooting process may also include:

[0114] 809. Continue filming? This step receives the user's instruction on whether to continue filming. If filming is not to continue, the filming process ends.

[0115] 810. If shooting continues, output a reminder message suggesting that the user change the shooting angle or shoot at a different time.

[0116] 811. Check if the shooting angle of the detection equipment has changed.

[0117] 812. Check if the preset time has been exceeded.

[0118] If the shooting angle of the device is changed, or if the shooting angle of the device is not changed but the continuous dwell time exceeds the preset time, continue to step 801.

[0119] In some embodiments, in step 803, the ratio of the number of edge pixels (pixel intensity 255 or 1) in the reflective edge image to the number of edge pixels in the first edge image is calculated to obtain the reflectance level in the color image.

[0120] In some embodiments, the first preset value in step 803 is 0.75.

[0121] As shown in the table below, experimental studies performed on multiple images show that the edge pixel count ratio is less than 0.4 for very faint reflective patterns (Figure 1(b)), and between 0.4 and 0.75 for faint reflective patterns (Figure 1(c)). The edge pixel count ratio is greater than 0.75 for thick reflective patterns (Figure 1(d)).

[0122] Edge pixel ratio Reflection level Less than 0.4 Very light 0.4～0.75 light Greater than 0.75 thick

[0123] If the reflection pattern determines that the reflection level is greater than or equal to a first preset value (i.e., the reflection level is high reflection level), the electronic device will send a message to the user indicating that the image is significantly affected by reflection, asking whether to save the image. It may also output a reminder message suggesting changing the shooting angle or taking the picture at a different time. The user can choose to change the shooting angle. If the shooting angle is changed, the earlier image (i.e., the color image and depth image mentioned above) will be deleted from the cache memory. Finally, when the reflection level is acceptable (less than the first preset value), the image is saved to the device's memory.

[0124] The above shooting process, except Figure 2 In addition to the electronic device shown, this technology is also applicable to cameras, camcorders, tablets, and any other similar devices that have a color camera sensor and a depth sensor. Optionally, the device may have a display device and a processor.

[0125] The imaging method described in this application distinguishes between transmission and reflection patterns from edges detected in captured color and depth images, and then provides a threshold to estimate the degree of reflection patterns in the scene. This provides users with an option to consider image quality before shooting, thereby achieving good photographic results and saving storage space.

[0126] like Figure 10 As shown, embodiments of this application also provide an image processing apparatus 50, including;

[0127] Acquisition unit 53 is used to acquire source images;

[0128] Recognition unit 54 is used to recognize reflection patterns in the source image;

[0129] The reflection level acquisition unit 51 is used to generate a reflection level that characterizes the strength of the reflection pattern based on the identified reflection pattern, wherein the reflection pattern is an image formed in the source image due to the reflection phenomenon.

[0130] Storage unit 52 is used to store the source image when the reflection level is less than or equal to a first preset value.

[0131] In some embodiments, such as Figure 11 As shown, the identification unit 54 includes:

[0132] The calibration module is used to perform field-of-view matching and pixel correspondence matching processing on the color image and the depth image, so that the processed color image and the depth image have the same field of view and have a one-to-one pixel correspondence.

[0133] An illuminance normalization module is used to perform illuminance normalization processing on the color image after field-of-view matching and pixel correspondence matching processing, so as to correct the illuminance unevenness of the color image.

[0134] A color image detection module is used to perform edge detection on the color image and generate a first edge image based on the edge detection results;

[0135] A hole-filling module is used to fill holes in the depth image after field-of-view matching and pixel correspondence matching processing to optimize the quality of the depth image.

[0136] The depth segmentation module is used to perform depth segmentation on the depth image and generate a first image that includes multiple grayscale ranges;

[0137] A multi-level thresholding processing module is used to perform multi-level thresholding processing on the first image to generate multiple binary images;

[0138] A color image detection module is used to perform edge detection on each of the plurality of binary images and generate a second edge image based on the edge detection results.

[0139] The reflection level acquisition unit 51 includes:

[0140] The edge matching module is used to perform edge matching on the first edge image and the second edge image to filter out pixels that are both edges in the first edge image and the second edge image, and generate a common edge image based on the filtering results.

[0141] The reflection level calculation module is used to calculate the reflection level based on the common edge image and the first edge image.

[0142] The implementation of each module's function can be referred to the corresponding steps in the image processing method described above, and will not be repeated here.

[0143] The electronic device of this application includes, but is not limited to, cameras, mobile phones, tablet computers, mobile computers, laptops, e-book players, etc., and the electronic device has at least one camera sensor and at least one depth sensor. Before saving the source image, the electronic device predicts the quality of the captured image based on the reflectance level. The source image is only saved if the reflectance level is less than or equal to a first preset value; otherwise, the source image is deleted. This provides the user with an option to consider image quality before shooting, thereby achieving good photographic results and saving storage space.

[0144] like Figure 12As shown, embodiments of this application also provide an electronic device 40, including a memory 401 and a processor 402. The memory 401 stores a program that is executed by the processor to perform the image processing method described in any of the above embodiments.

[0145] In some embodiments, the electronic device of this application can be a terminal device having a camera sensor, a depth sensor, a storage medium, and a display, such as a mobile phone. The camera sensor is used to acquire color images, and the depth sensor is used to acquire depth images. The processor 402 performs image processing on the color images and depth images to obtain or generate the reflectance level of the color images, and determines whether to store them. For reflectance levels less than or equal to a first preset value, they can be stored directly; for reflectance levels greater than the first preset value, the reflectance level is output on the display, and the user determines whether to store it, avoiding the storage of low-quality images with high reflectance levels. In this way, it provides the user with an option to consider image quality before shooting, thereby achieving good photographic results and saving storage space.

[0146] Embodiments of this application also provide a readable storage medium storing a program for being executed by a processor to perform the image processing method as described in any of the preceding claims.

[0147] Although this application has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and drawings. This application includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components, the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this specification shown herein.

[0148] That is, the above description is only an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made using the content of this application’s specification and drawings, such as the combination of technical features between different embodiments, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of this application.

[0149] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as more preferred or advantageous than other embodiments. This application has been provided above to enable any person skilled in the art to implement and use it. Various details have been set forth in the above description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other embodiments, well-known structures and processes will not be described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

Claims

1. An image processing method, characterized in that, include: Acquire source images captured by a camera device, the source images including color images and depth images captured by the camera device of the same scene; The process involves identifying reflection patterns in a source image and generating a reflection level characterizing the strength of these patterns. The reflection pattern is the image formed in the source image by reflection from an object outside the shooting range. Generating the reflection level characterizes the strength of the reflection pattern by the identified reflection pattern includes: performing field-of-view matching and pixel-to-pixel matching on the color image and the depth image to ensure that the processed color image and the depth image have the same field of view and a one-to-one pixel correspondence; performing edge detection on the color image and generating a first edge image based on the edge detection results; performing edge detection on the depth image and generating a second edge image based on the edge detection results; performing edge matching on the first edge image and the second edge image to filter out pixels that appear as edges in both the first and second edge images, and generating a common edge image based on the filtering results; and determining the reflection level based on the common edge image and the first edge image. The edges of the color image are points in the color image where the image intensity value is discontinuously greater than a predetermined threshold, and the edges of the depth image are points in the depth image where the image intensity value is discontinuously greater than a predetermined threshold. When the reflection level is less than or equal to a first preset value, the source image is saved in the memory; When the reflection level is greater than the first preset value, the reflection level is output; Based on the user's input instruction to store or discard the storage, the source image is deleted or saved in the memory.

2. The image processing method according to claim 1, characterized in that, Determining the reflection level based on the common edge image and the first edge image includes: Obtain pixel information that appears as an edge in the first edge image but does not appear as an edge in the common edge image; A reflection edge image is generated based on the acquired pixel information; The reflection level is obtained by determining the ratio of the number of pixels that are edges in the reflected edge image to the number of pixels that are edges in the first edge image.

3. The image processing method according to claim 1, characterized in that, After performing field-of-view matching and pixel-to-pixel matching on the color image and the depth image, and before performing edge detection on the color image, the image processing method further includes: The color image is subjected to illuminance normalization processing to make the illuminance of the color image uniform.

4. The image processing method according to claim 1, characterized in that, After performing field-of-view matching and pixel-to-pixel matching on the color image and the depth image, and before performing edge detection on the depth image, the image processing method further includes: Hole filling is applied to the depth image to optimize its quality.

5. The image processing method according to any one of claims 1-4, characterized in that, The steps of performing edge detection on the depth image and generating a second edge image based on the edge detection results include: The depth image is divided into depths based on pixel depth values ​​to generate a first image including multiple grayscale ranges; The first image is subjected to multi-level thresholding to generate multiple binary images, wherein each binary image corresponds to a grayscale range of the first image. Edge detection is performed on each of the plurality of binary images, and the second edge image is generated based on the edge detection results.

6. The image processing method according to claim 5, characterized in that, The step of dividing the depth image according to pixel depth values ​​to generate a first image including multiple grayscale ranges includes: Histogram analysis is performed on the depth image to obtain an intensity histogram, and an arbitrary peak detection algorithm is applied to the obtained intensity histogram to obtain the first image including multiple grayscale ranges.

7. The image processing method according to claim 1, characterized in that, Before identifying the reflection pattern in the source image and generating a reflection level characterizing the strength of the reflection pattern based on the identified reflection pattern, the image processing method further includes: acquiring the source image and storing the source image in a buffer memory; The step of saving the source image in the memory when the reflection level is less than or equal to the first preset value is: when the reflection level is less than or equal to the first preset value, the source image is transferred from the buffer memory to the non-buffer memory; The step of deleting or storing the source image in the memory according to the user's input instruction to store or abandon storage is: deleting the source image from the buffer memory or transferring the source image from the buffer memory to the non-buffered memory according to the user's input instruction to store or abandon storage.

8. The image processing method according to claim 1 or 7, characterized in that, When the reflectance level is greater than the first preset value, the image processing method further includes: Output a reminder message to prompt the user to change the shooting angle and reshoot.

9. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire source images, which include color images and depth images of the same scene acquired by the camera device; The recognition unit is used to recognize the reflection pattern in the source image; A reflection level acquisition unit is used to generate a reflection level characterizing the strength of the reflection pattern based on the identified reflection pattern, wherein the reflection pattern is the image formed in the source image by the reflection phenomenon of an object outside the shooting range. A storage unit is used to store the source image when the reflection level is less than or equal to a first preset value, and to output the reflection level when the reflection level is greater than the first preset value. Based on the user's input instruction to store or discard the storage, the source image is deleted or saved in the memory; The identification unit includes: The calibration module is used to perform field-of-view matching and pixel correspondence matching processing on the color image and the depth image, so that the processed color image and the depth image have the same field of view and have a one-to-one pixel correspondence. A color image detection module is used to perform edge detection on the color image and generate a first edge image based on the edge detection results; and to perform edge detection on the depth image and generate a second edge image based on the edge detection results. The edges of the color image are points in the color image where the image intensity value is discontinuously greater than a predetermined threshold, and the edges of the depth image are points in the depth image where the image intensity value is discontinuously greater than a predetermined threshold. The edge matching module is used to perform edge matching on the first edge image and the second edge image to filter out pixels that are both edges in the first edge image and the second edge image, and generate a common edge image based on the filtering results. The reflection level calculation module is used to calculate the reflection level based on the common edge image and the first edge image.

10. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a program for being executed by the processor to perform the image processing method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that, The readable storage medium stores a program that is executed by a processor to perform the image processing method as described in any one of claims 1 to 8.

Images