Image processing method and related device

Abstract

The present application provides an image processing method and related devices. The electronic device can capture a first image and a second image through a camera. The electronic device can identify an overexposed area from the first image. The electronic device can identify the area where the first moving object is located in the first image based on the first image and the second image. If the overexposed area does not overlap with the area where the first moving object is located in the first image, the electronic device can deblur the area where the first moving object is located in the first image to obtain a third image, and display the third image on the camera shooting interface. If the overexposed area overlaps with the area where the first moving object is located, the electronic device does not deblur the area where the first moving object is located in the first image, and displays the first image on the camera shooting interface. In this way, the clear details in the originally blurred image can be restored to improve the shooting quality.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and related devices. Background Art

[0002] During the photo-taking process, motion blur occurs due to the relative motion between the imaging device and the moving scene objects, and the obtained image loses important detail information. Generally, in order to restore the detail information of the image, deblurring technology is used to eliminate motion blur and restore the original detail information of the moving scene objects. However, the existing deblurring technology is not very effective in restoring the detail information of the image. Summary of the invention

[0003] The present application provides an image processing method and related devices, which can restore clear details in originally blurred images and improve shooting quality.

[0004] In a first aspect, the present application provides an image processing method, including: an electronic device can capture a first image and a second image through a camera, wherein the first image and the second image are adjacent in the image stream acquired by the camera. The electronic device can identify an overexposed area from the first image. The electronic device can identify the area where the first moving object is located in the first image based on the first image and the second image. If the overexposed area does not overlap with the area where the first moving object is located in the first image, the electronic device can deblur the area where the first moving object is located in the first image to obtain a third image, and display the third image on the camera shooting interface. The clarity of the first moving object in the third image is greater than the clarity of the first moving object in the first image. If the overexposed area overlaps with the area where the first moving object is located, the electronic device does not deblur the area where the first moving object is located in the first image, and displays the first image on the camera shooting interface.

[0005] Through an image processing method provided by the present application, an electronic device can determine whether an overexposed area in a first image and an area where a first moving object is located overlap. If the overexposed area and the area where the first moving object is located in the first image do not overlap, the electronic device can deblur the area where the first moving object is located in the first image. If the overexposed area and the area where the first moving object is located overlap, the electronic device does not deblur the area where the first moving object is located in the first image. In this way, the electronic device can determine whether it is necessary to restore clear details in the originally blurred image and improve the shooting quality by determining whether the overexposed area and the area where the first moving object is located in the first image overlap.

[0006] In one possible implementation, an overexposed area is identified from a first image, specifically comprising: grayscale processing and downsampling the first image to obtain a fourth image. The grayscale values ​​of pixels in the fourth image whose grayscale values ​​are greater than or equal to a first threshold are adjusted to an upper grayscale limit value. The grayscale values ​​of pixels in the fourth image whose grayscale values ​​are less than the first threshold are adjusted to a lower grayscale limit value to obtain a fifth image. Noise points in the fifth image are removed to obtain a sixth image. An overexposed area is identified from the sixth image. In this way, grayscale processing and downsampling can reduce the image size, reduce the computational complexity of subsequent networks, and thus improve computational efficiency. At the same time, removing noise points can remove interference information, making the identification of overexposed areas more accurate.

[0007] In a possible implementation manner, removing noise points in the fifth image to obtain the sixth image specifically includes: removing noise points in the fifth image by performing an opening operation to obtain the sixth image.

[0008] In a possible implementation, based on the first image and the second image, the area where the first moving object is located in the first image is identified, specifically including: grayscale processing and downsampling the second image to obtain a seventh image. Image registration is performed on the fourth image and the seventh image, and a differential operation is performed based on the registered fourth image and seventh image to obtain an eighth image. A segmentation map is generated based on the eighth image to obtain a ninth image. Noise points in the ninth image are removed to obtain a tenth image. The area where the first moving object is located is identified from the tenth image. In this way, grayscale processing and downsampling can reduce the image size, reduce the computational complexity of the subsequent network, and thus improve the computational efficiency. At the same time, removing noise points can remove interference information, making the identification of overexposed areas more accurate.

[0009] In a possible implementation, generating a segmented image based on the eighth image to obtain a ninth image specifically includes: dividing the eighth image into a plurality of windows, and calculating the grayscale average value of pixels in each window. Determining one or more first windows whose grayscale average value is greater than or equal to a second threshold, and one or more second windows whose grayscale average value is less than the second threshold. Adjusting the grayscale value of the pixel in the first window in the eighth image to the grayscale upper limit value, and adjusting the grayscale value of the pixel in the second window to the grayscale lower limit value, to obtain the ninth image.

[0010] In a possible implementation manner, removing noise points in the ninth image to obtain the tenth image specifically includes: removing noise points in the ninth image by performing an opening operation to obtain the tenth image.

[0011] In a possible implementation, based on the first image and the second image, identifying the region where the first moving object is located in the first image specifically includes: determining optical flow information between the first image and the second image, where the optical flow information is used to indicate the motion vector of each pixel between the first image and the second image. Determining the region where the first moving object is located in the first image based on the optical flow information. In this way, by determining the region where the first moving object is located in the first image through the optical flow information, the region where the first moving object is located can be accurately identified without knowing any information about the scene.

[0012] In a possible implementation, based on the first image and the second image, identifying the area where the first moving object is located in the first image specifically includes: acquiring first depth information corresponding to the first image and second depth information corresponding to the second image. Based on the first image, the second image, the first depth information, and the second depth information, determining the motion field vector information between the first image and the second image. Based on the motion field vector information, determining the area where the first moving object is located in the first image. In this way, by determining the area where the first moving object is located in the first image through the motion field vector information, the recognition accuracy of the area where the first moving object is located can be improved.

[0013] In a second aspect, the present application provides an electronic device comprising a camera, one or more processors and one or more memories; wherein the camera, the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the method in any possible implementation of the above-mentioned first aspect is executed.

[0014] In a third aspect, the present application provides another electronic device, including a motion area detection module, an overexposed area detection module, a judgment module and a deblurring module, wherein the motion area detection module, the overexposed area detection module, the judgment module and the deblurring module are used to execute the method in any possible implementation of the first aspect mentioned above.

[0015] In a fourth aspect, the present application provides a chip system, comprising a processing circuit and an interface circuit, wherein the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method in any possible implementation of any of the above aspects.

[0016] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a processor of an electronic device, enables execution of a method in any possible implementation of the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a software system architecture of an electronic device provided in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of a camera usage scenario provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a method flow for image processing provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of an image acquisition method provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of image grayscale processing and downsampling provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of converting a RAW image in RGGB format provided in an embodiment of the present application into a grayscale image;

[0024] Figure 8 A schematic diagram of generating an exposure image, removing noise points, and identifying overexposed areas provided in an embodiment of the present application;

[0025] Fig. 9 A schematic diagram of another image grayscale processing and downsampling provided in an embodiment of the present application;

[0026] Fig.10 A schematic diagram of performing image registration and differential operations on two images provided in an embodiment of the present application;

[0027] Fig.11 A schematic diagram of generating a segmented image provided in an embodiment of the present application;

[0028] Fig.12 A schematic diagram of generating a segmented image from an eighth image provided in an embodiment of the present application;

[0029] Fig.13 A schematic diagram of removing noise points and identifying the area where the first moving object is located provided in an embodiment of the present application;

[0030] Figure 14A-14B A set of schematic diagrams for determining whether an overexposed area and an area where a first moving object is located overlap is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0032] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0033] The following introduces the hardware structure of an electronic device provided in an embodiment of the present application.

[0034] Figure 1 A schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application is shown.

[0035] It should be understood that Figure 1 The electronic device 100 shown is only an example, and the electronic device 100 may have more Figure 1 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0036] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K, etc.

[0037] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0038] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0039] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0040] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0041] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0042] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0043] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0044] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0045] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0046] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0047] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0048] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0049] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0050] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0051] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0052] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0053] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0054] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0055] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0056] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0057] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0058] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0059] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0060] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0061] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0062] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0063] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0064] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0065] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0066] The button 190 includes a power button, a volume button, etc. The button 190 may be a mechanical button. It may also be a touch button. The electronic device 100 may receive a button input and generate a key signal input related to the user settings and function control of the electronic device 100. The motor 191 may generate a vibration prompt. The SIM card interface 195 is used to connect a SIM card. The SIM card may be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or pulling the SIM card interface 195 out.

[0067] In the embodiment of the present application, the device type of the electronic device 100 can be any one of a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), vehicle-mounted devices or smart city devices, etc.

[0068] The following introduces a software system architecture of an electronic device 100 provided in an embodiment of the present application.

[0069] Figure 2A schematic diagram of a software system architecture of an electronic device 100 provided in an embodiment of the present application is shown.

[0070] like Figure 2 As shown, the software system architecture of the electronic device 100 may include an application (APP) layer, a native service (native) layer, and a kernel (kernel) layer.

[0071] The kernel layer may include a camera driver 201 . The native layer may include a motion region detection module 202 , an overexposed region detection module 203 , a judgment module 204 , a deblurring module 205 , etc. The application layer may include a camera application 206 .

[0072] The following is a detailed description of the processing flow in the embodiment of the present application in conjunction with the software system architecture diagram of the camera on the electronic device 100:

[0073] 1. Camera driver 201 can turn the camera ( Figure 2 The first image and the second image acquired (not shown) are uploaded to the motion area detection module 202.

[0074] 2. The motion region detection module 202 may identify the region where the first motion object is located in the first image based on the first image and the second image.

[0075] 3. The motion region detection module 202 may upload the region where the first motion object is located in the first image to the determination module 204 .

[0076] 4. Camera driver 201 can turn the camera ( Figure 2 The first image obtained by the processor (not shown in the figure) is uploaded to the overexposed area detection module 203.

[0077] 5. The overexposed area detection module 203 can identify the overexposed area from the first image.

[0078] 6. The overexposed area detection module 203 may upload the overexposed area in the first image to the determination module 204 .

[0079] 7. The determination module 204 may determine whether the area where the first moving object is located in the first image and the overexposed area in the first image overlap.

[0080] 8. If the area where the first moving object is located in the first image overlaps with the overexposed area in the first image, the determination module 204 may upload the first image to the camera application 206. The camera application 206 may display the first image in a camera preview frame.

[0081] 9. If the area where the first moving object is located in the first image and the overexposed area in the first image do not overlap, the determination module 204 may send the first image, the second image and the area where the first moving object is located in the first image to the deblurring module 205 .

[0082] 10. The deblurring module 205 may perform deblurring processing on the area where the first moving object is located in the first image to obtain a third image.

[0083] Among them, after receiving the first image, the second image and the area where the first moving object is located in the first image, the deblurring module 205 can use the details in the second image to deblur the area where the first moving object is located in the first image through a deblurring network of multiple frames of images, thereby restoring more details of the area where the first moving object is located in the first image to improve the clarity of the area where the first moving object is located in the first image.

[0084] In a possible implementation, if the area where the first moving object is located in the first image and the overexposed area in the first image do not overlap, the judgment module 204 may directly send the first image and the area where the first moving object is located in the first image to the deblurring module 205. After receiving the first image and the area where the first moving object is located in the first image, the deblurring module 205 may perform a deblurring process on the area where the first moving object is located in the first image through a deblurring network of a single-frame image, thereby restoring more details of the area where the first moving object is located in the first image, so as to improve the clarity of the area where the first moving object is located in the first image.

[0085] 11. The deblurring module 205 may upload the third image to the camera application 206. The camera application 206 may display the third image in a camera preview box.

[0086] The following introduces a camera usage scenario in an embodiment of the present application.

[0087] Figure 3 A camera usage scenario in an embodiment of the present application is shown.

[0088] For example, Figure 3 As shown, the electronic device 100 can display a camera shooting interface 320. The camera shooting interface 320 can include a function selection area 326, a camera preview frame 328, a mode selection area 324 and a quick function area 329.

[0089] The camera preview frame 328 may include an image 327. The image 327 may be a frame of an image in an image stream obtained by the electronic device 100 through a camera. The image 327 may include a first moving object 3271 and an overexposed area 3272. The first moving object 3271 may be an object in motion, and the overexposed area 3272 may be an area with too much light. When the electronic device 100 and the first moving object 3271 move relative to each other, the first moving object 3271 may become blurred. When the first moving object 3271 does not overlap with the overexposed area 3272, the electronic device 100 may deblur the first moving object 3271. When the first moving object 3271 overlaps with the overexposed area 3272, the electronic device 100 does not deblur the first moving object 3271.

[0090] The function selection area 326 may include one or more function controls, such as a smart function control 326A, an artificial intelligence function control 326B, a flash function control 326C, and a setting function control 326D.

[0091] The mode selection area 324 may include one or more camera mode controls, such as a large aperture mode control 324A, a night scene mode control 324B, a portrait mode control 324C, a photo mode control 324D, a video mode control 324E, and a professional mode control 324F.

[0092] The quick function area 329 may include a gallery control 321, a shutter control 322, and a front and rear camera conversion control 323, etc. The electronic device 100 may enter the gallery based on the gallery control 321. The electronic device 100 may take photos, record videos, etc. based on the shutter control 322. The electronic device 100 may switch between the front camera and the rear camera based on the front and rear camera conversion control 323.

[0093] When the background of the first moving object 3271 in the image 327 is the overexposed area 3272, since the brightness of the overexposed area 3272 is too strong, erroneous information will be restored in restoring the motion blur of the first moving object 3271. Therefore, an image processing method is provided in an embodiment of the present application. When the electronic device 100 recognizes that the area where the first moving object 3271 in the image 327 overlaps with the overexposed area 3272, deblurring may not be performed. When the electronic device 100 recognizes that the area where the first moving object 3271 in the image 327 does not overlap with the overexposed area 3272, deblurring is performed on the area where the first moving object is located.

[0094] The following is a flow chart of an image processing method involved in an embodiment of the present application.

[0095] Figure 4 A flow chart of an image processing method designed in an embodiment of the present application is shown.

[0096] like Figure 4 As shown, the image processing method comprises the following steps:

[0097] S401. Acquire a first image and a second image through a camera, wherein the first image and the second image are adjacent in an image stream acquired by the camera.

[0098] The electronic device 100 can obtain the image stream through the camera. Figure 5 As shown, when there is a first moving object 501 and an overexposed area 502 in the image of the image stream, the electronic device 100 can obtain the first image and the second image from the image stream. Among them, the first moving object 501 can be an object in motion. The overexposed area 502 can be an area with too bright light. The first image includes the first moving object 501 and the overexposed area 502. The second image also includes the first moving object 501 and the overexposed area 502. The second image can be the previous image frame of the first image or the next image frame of the first image. The format of the first image can be RAW format. The format of the second image can be RAW format.

[0099] S402: Perform grayscale processing and down-sampling on the first image to obtain a fourth image.

[0100] like Figure 6 As shown, the electronic device 100 can perform grayscale processing and downsampling on the first image to obtain a fourth image. The first image can be converted into a fourth image by grayscale processing and downsampling, which can reduce the size of the first image and reduce the complexity of subsequent calculations, thereby improving the operation efficiency.

[0101] For example, Figure 7 As shown, a RAW image in RGGB format is taken as an example. The electronic device 100 can use a 2×2 window to divide the RAW image in RGGB format. A 2×2 window just covers a set of RGGB: R (red channel), G1 (green channel), G2 (green channel), and B (blue channel). There is no overlap between windows. The electronic device 100 can calculate the average values ​​of the four channels in each window in turn, and then obtain an image composed of the average values ​​(P), that is, a grayscale image. Therefore, an average value in a grayscale image is also called a grayscale value.

[0102] In a RAW image in RGGB format, one channel (R / G / G / B) can be called a pixel. In a grayscale image, one grayscale value P can be called a pixel. Exemplarily, when the size of the initial RAW image in RGGB format is H1×W1, the size of the converted grayscale image can be H2×W2. Among them, H1=2×H2, W1=2×W2. Exemplarily, H1×W1=1920×1080, H2×W2=960×540. The size of the RAW image H1×W1 means that the RAW image has H1 pixels horizontally and W1 pixels vertically.

[0103] After obtaining the grayscale image, the electronic device 100 may downsample the grayscale image to obtain a downsampled image, and the size of the downsampled image is recorded as H3×W3. For example, when the electronic device 100 performs a 1 / 2 downsampling on the grayscale image, the size of the downsampled image is H3×W3=1 / 2 H2×1 / 2W2. Taking the above 960×540 grayscale image as an example, the size of the downsampled image after 1 / 2 downsampling is 480×270.

[0104] In some embodiments, the image format of the RAW image may be other formats such as RYYB format or RGBW format. Figure 7 The method shown processes RAW images in other formats such as RYYB format and RGBW format to obtain corresponding grayscale images and down-sampled images.

[0105] The downsampled image mentioned above may be referred to as a fourth image.

[0106] S403, adjusting the grayscale values ​​of pixels in the fourth image whose grayscale values ​​are greater than or equal to the first threshold to the grayscale upper limit value; adjusting the grayscale values ​​of pixels in the fourth image whose grayscale values ​​are less than the first threshold to the grayscale lower limit value, to obtain a fifth image.

[0107] For example, the first threshold value may be 235, and the grayscale range value may be 0-255. The grayscale upper limit value is 255, and the grayscale lower limit value is 0. Figure 8 As shown, the electronic device 100 can use the exposure threshold comparison method to generate an exposure image, which can be referred to as the fifth image. For example, the electronic device 100 can adjust the grayscale value of pixels in the fourth image whose grayscale value is greater than or equal to 235 to 255. The electronic device 100 can adjust the grayscale value of pixels in the fourth image whose grayscale value is less than 235 to 0 to obtain the fifth image. Among them, the pixel with a grayscale value of 255 is pure white, and the pixel with a grayscale value of 0 is pure black. In this way, the fourth image can be converted into the fifth image through the exposure threshold comparison method, which can make the overexposed area more prominent, thereby helping the electronic device to quickly complete the identification of the overexposed area.

[0108] S404: Remove noise points in the fifth image to obtain a sixth image.

[0109] The electronic device 100 may use the exposure threshold comparison method to generate an exposure image, and some noise may exist in the exposure image. The exposure image may be referred to as the fifth image. Figure 8 As shown, the fifth image may include noise points such as noise point 801 and noise point 802. Among them, the noise point includes a random change in brightness or color information in the image (which the photographed object itself does not have). The electronic device 100 can remove the noise points in the fifth image through an opening operation, and identify the overexposed area 803 to obtain a sixth image. Among them, the opening operation can remove small bright spots and burrs in the image. In this way, by removing the noise points in the fifth image to obtain the sixth image, the interference information in the fifth image can be removed, making it more accurate to identify the overexposed area in the sixth image.

[0110] S405: Identify an overexposed area from the sixth image.

[0111] like Figure 8 As shown, the electronic device 100 can identify the overexposed area 803 from the sixth image.

[0112] S406: Perform grayscale processing and down-sampling on the second image to obtain a seventh image.

[0113] like Fig. 9 As shown, the electronic device 100 can perform grayscale processing and down-sampling on the second image to obtain a seventh image. For details, please refer to the above Figure 6 The above is not elaborated here.

[0114] S407 , performing image registration on the fourth image and the seventh image, and performing a difference operation based on the registered fourth image and the seventh image to obtain an eighth image.

[0115] like Fig.10 As shown, the electronic device 100 can perform image registration on the fourth image and the seventh image, and perform a differential operation based on the registered fourth image and the seventh image to obtain an eighth image. By performing image registration and differential operation on the fourth image and the seventh image, the features of the first moving object 1101 in the eighth image can be identified.

[0116] The electronic device 100 may map the fourth image onto the seventh image so that pixels corresponding to the same spatial position of the fourth image and the seventh image correspond one to one. The electronic device 100 may subtract the grayscale value of the pixel corresponding to the same spatial position in the seventh image from the grayscale value of the pixel in the fourth image to obtain the eighth image.

[0117] S408 . Generate a segmented image based on the eighth image to obtain a ninth image.

[0118] like Fig.11 As shown, since the features of the first moving object 1101 in the eighth image are not clear, the electronic device 100 can generate a segmented image based on the eighth image to obtain a ninth image. The features of the first moving object 1102 that is clear in the ninth image will be clearer. In this way, by enhancing the unclear features in the eighth image to obtain the ninth image, the features of the first moving object 1102 that is clear in the ninth image can be made clearer, thereby helping the electronic device 100 to quickly complete the recognition of the clear first moving object 1102.

[0119] The electronic device 100 may divide the eighth image into a plurality of windows and calculate the grayscale average of the pixels in each window. One or more first windows whose grayscale average is greater than or equal to the second threshold value and one or more second windows whose grayscale average is less than the second threshold value are determined. The grayscale value of the pixel in the first window of the eighth image is adjusted to the grayscale upper limit value, and the grayscale value of the pixel in the second window is adjusted to the grayscale lower limit value, to obtain a ninth image.

[0120] For example, Fig.12 As shown, the second threshold value can be 225, and the grayscale range value can be 0-255. Among them, the grayscale upper limit value is 255, and the grayscale lower limit value is 0. The electronic device 100 can use a 2×2 window to divide the eighth image. A 2×2 window covers 4 pixels. There is no overlap between windows. Among them, the eighth image includes multiple pixels, and P1, P2,..., P63 and P64 represent the grayscale values ​​of the pixels. The electronic device 100 can calculate the average value of the grayscale values ​​of the four pixels in each window in turn, and then obtain an image composed of the average values, that is, a grayscale average image. For example, the grayscale values ​​of four pixels are P1, P2, P9 and P10, respectively, and the average value of the grayscale values ​​of the four pixels is P101.

[0121] The electronic device 100 can use a window with a grayscale value greater than or equal to 225 in the grayscale average image as a first window, and adjust the grayscale value of the pixel in the first window to 255. The electronic device 100 can use a window with a grayscale value less than 225 in the grayscale average image as a second window, and adjust the grayscale value of the pixel in the second window to 0 to obtain a segmented image. The grayscale average image includes multiple windows, and P101, P102, ..., P115 and P116 represent the grayscale average value of the pixels in each window. For example, if the grayscale average values ​​P104, P105, P107, P109 and P111 in the grayscale average image are greater than 225, the electronic device 100 can adjust the grayscale values ​​of the pixels in the P104, P105, P107, P109 and P111 windows to 255. For the remaining grayscale average values ​​less than 255 in the grayscale average image (for example, P01, P102, and P103), the electronic device 100 can adjust the grayscale values ​​of the pixels in the window with the grayscale average value less than 255 in the grayscale average image to 0, and obtain a segmented image. Among them, the pixels with a grayscale value of 255 are pure white, and the pixels with a grayscale value of 0 are pure black.

[0122] S409: Remove noise points in the ninth image to obtain a tenth image.

[0123] like Fig.13 As shown, when the clarity of the clear first moving object 1102 in the ninth image is enhanced, the clarity of the noise points (for example, noise points 1103 and noise points 1104) in the ninth image is also enhanced. The electronic device 100 can remove the noise points in the ninth image through an opening operation, and identify the area 1105 where the first moving object is located, to obtain the tenth image.

[0124] S410: Identify a region where a first moving object is located from a tenth image.

[0125] like Fig.13 As shown, the electronic device 100 can identify the area 1105 where the first moving object is located from the tenth image.

[0126] S411 , determining whether the overexposed area and the area where the first moving object is located overlap in the first image.

[0127] The electronic device 100 may determine whether the area 1105 where the first moving object is located and the overexposed area 803 in the first image overlap based on the overexposed area 803 in the sixth image and the area 1105 where the first moving object is located in the tenth image.

[0128] S412: If the overexposed area and the area where the first moving object is located do not overlap in the first image, deblur the area where the first moving object is located to obtain a third image.

[0129] If the area 1105 where the first moving object is located and the overexposed area 803 do not overlap in the first image (for example Fig.14A As shown in FIG. 1 ), the electronic device 100 can perform a deblurring process on the region 1105 where the first moving object is located in the first image to obtain a third image, and display the third image on the camera shooting interface. The clarity of the first moving object in the third image is greater than the clarity of the first moving object in the first image. In this way, the detailed information of the image can be restored by deblurring the region 1105 where the first moving object is located in the first image.

[0130] In one possible implementation, the electronic device 100 may input the first image and the area where the first moving object in the first image is located into a deblurring network of a single-frame image, and deblur the area where the first moving object in the first image is located through the deblurring network of the single-frame image to obtain a third image.

[0131] The electronic device 100 may convert the first image from RAW format to RGB format. The electronic device 100 may identify the region where the first moving object is located in the first image after conversion to RGB format. The electronic device 100 may perform a deblurring process on the region where the first moving object is located in the first image after conversion to RGB format to obtain a third image.

[0132] In one possible implementation, the electronic device 100 may send the first image, the second image, and the area where the first moving object in the first image is located into a multi-frame image deblurring network, and through the multi-frame image deblurring network, use the details in the second image to deblur the area where the first moving object in the first image is located, thereby restoring more details of the area where the first moving object in the first image is located, so as to improve the clarity of the area where the first moving object in the first image is located.

[0133] Among them, the electronic device 100 can convert the first image and the second image from the RAW format to the RGB format. The electronic device 100 can identify the blurred area of ​​the first moving object in the first image after conversion to the RGB format. The electronic device 100 can determine the clear area corresponding to the blurred area from the second image after conversion to the RGB format, and backfill the features of the clear area to the blurred area of ​​the first moving object in the first image after conversion to the RGB format, to obtain a third image, the clarity of the clear area is higher than the clarity of the blurred area.

[0134] S413: If the overexposed area and the area where the first moving object is located overlap in the first image, do not perform deblurring on the area where the first moving object is located.

[0135] If the area 1105 where the first moving object is located and the overexposed area 803 overlap in the first image (for example Fig. 14B As shown), the electronic device 100 does not deblur the area 1105 where the first moving object is located, and displays the first image on the camera shooting interface.

[0136] In a possible implementation, the electronic device 100 may determine optical flow information between the first image and the second image. The optical flow information is used to indicate the motion vector of each pixel between the first image and the second image. The electronic device 100 may determine the region where the first moving object is located in the first image based on the optical flow information. In this way, by determining the region where the first moving object is located in the first image through the optical flow information, the region where the first moving object is located can be accurately identified without knowing any information about the scene.

[0137] In a possible implementation, the electronic device 100 may obtain first depth information corresponding to the first image and second depth information corresponding to the second image. The electronic device 100 may determine the motion field vector information between the first image and the second image based on the first image, the second image, the first depth information, and the second depth information. The electronic device 100 may determine the region where the first moving object is located in the first image based on the motion field vector information. In this way, by determining the region where the first moving object is located in the first image through the motion field vector information, the recognition accuracy of the region where the first moving object is located may be improved.

[0138] In one possible implementation, the first moving object may be a certain type of object (e.g., a person). For example, the electronic device 100 may identify the area where the moving person is located and the overexposed area in the first image. If the overexposed area does not overlap with the area where the moving person is located in the first image, the area where the moving person is located in the first image is deblurred to obtain a third image, and the third image is displayed on the camera shooting interface. If the overexposed area overlaps with the area where the moving person is located in the first image, the area where the moving person is located in the first image is not deblurred, and the first image is displayed on the camera shooting interface. In this way, the electronic device 100 can only identify a certain type of object in the image, and can deblur the certain type of object in a targeted manner.

[0139] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0140] The embodiments of the present application also provide a computer program product. When the computer program product is executed on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0141] The embodiment of the present application also provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0142] The term "user interface (UI)" in the specification and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the electronic device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as . A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0143] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0144] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0145] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0146] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized in that: include: A first image and a second image are captured by a camera, wherein the first image and the second image are adjacent to each other in an image stream acquired by the camera, and the first image and the second image are RAW images in RGGB format; Dividing the first image in RGGB format using a 2×2 sized window, wherein a 2×2 sized window covers a set of RGGB channels; Calculate the average value of the RGGB channel in each 2×2 window to obtain a grayscale image consisting of the average values; Down-sampling the grayscale image by 1 / 2 to obtain a fourth image; The grayscale values ​​of pixels in the fourth image whose grayscale values ​​are greater than or equal to the first threshold are adjusted to 255; the grayscale values ​​of pixels in the fourth image whose grayscale values ​​are less than the first threshold are adjusted to 0, thereby obtaining a fifth image; removing noise points from the fifth image to obtain a sixth image; identifying an overexposed area from the sixth image; Performing grayscale processing and downsampling on the second image to obtain a seventh image; Performing image registration on the fourth image and the seventh image, and performing a difference operation based on the registered fourth image and the seventh image to obtain an eighth image; Dividing the eighth image using windows of 2×2 pixels, and calculating the grayscale averages of pixels in each window of 2×2 pixels, wherein one window of 2×2 pixels includes channels of four adjacent different pixels in the eighth image, and windows of different 2×2 pixels do not overlap; Determine one or more first windows in which the grayscale average value is greater than or equal to a second threshold value, and one or more second windows in which the grayscale average value is less than the second threshold value, among the plurality of 2×2 pixel windows; Adjusting the grayscale values ​​of the pixels in the first window in the eighth image to 255, and adjusting the grayscale values ​​of the pixels in the second window to 0, to obtain a ninth image; removing noise points from the ninth image to obtain a tenth image; identifying a region where a first moving object is located from the tenth image; If the overexposed area does not overlap with the area where the first moving object is located in the first image, deblurring the area where the first moving object is located in the first image to obtain a third image, and displaying the third image on a camera shooting interface, wherein the clarity of the first moving object in the third image is greater than the clarity of the first moving object in the first image; If the overexposed area overlaps with the area where the first moving object is located, the area where the first moving object is located in the first image is not deblurred, and the first image is displayed on the camera shooting interface.

2. The method according to claim 1, characterized in that The removing noise from the fifth image to obtain the sixth image specifically includes: The noise points in the fifth image are removed by an opening operation to obtain a sixth image.

3. The method according to claim 1, characterized in that The removing of noise from the ninth image to obtain the tenth image specifically includes: The noise points in the ninth image are removed by an opening operation to obtain a tenth image.

4. The method according to claim 1, characterized in that The deblurring the area where the first moving object is located in the first image to obtain the third image specifically includes: converting the first image and the second image from a RAW format to an RGB format; identifying a blurred area of ​​the first moving object in the first image after conversion to the RGB format; A clear area corresponding to the blurred area is determined from the second image after conversion into the RGB format, and features of the clear area are backfilled into the blurred area of ​​the first moving object in the first image after conversion into the RGB format to obtain the third image, wherein the clarity of the clear area is higher than that of the blurred area.

5. An electronic device, characterized in that: It comprises a camera, one or more processors and one or more memories; wherein the camera, the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method as described in any one of claims 1 to 4 is executed.

6. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes a processing circuit and an interface circuit, the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a processor of an electronic device, cause the method according to any one of claims 1 to 4 to be executed.

Images