Image processing method and device, computer readable storage medium and electronic device

By acquiring reference images and images to be processed at different focus distances, image optimization technology is used to eliminate or reduce the impact of abnormality in protective cover on image quality, solving the problem of image quality degradation caused by abnormality in protective cover and achieving improvement in image quality.

CN114866699BActive Publication Date: 2024-11-19GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210562945.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-23
Publication Date
2024-11-19
Estimated Expiration
2042-05-23

AI Technical Summary

Technical Problem

Abnormalities in the protective cover (such as dirt, scratches, etc.) will lead to a decline in image quality, and it is difficult for the existing technology to effectively solve this problem.

Method used

By acquiring reference images and pending images captured based on different focus distances, the to-process images are optimized using the reference images to eliminate or reduce the impact of protective cover abnormalities on the image.

Benefits of technology

Improved image quality without the need for additional hardware, suitable for various end devices, and can reduce or eliminate the impact of protective cover abnormalities on the image to a certain extent.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114866699B_ABST
    Figure CN114866699B_ABST
Patent Text Reader

Abstract

The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, and relates to the field of image processing technology. The image processing method comprises: obtaining a reference image, the reference image is obtained by shooting based on a first focal distance, and the first focal distance is determined according to the distance between the protective cover of the first lens and the first lens; obtaining an image to be processed shot by the first lens based on a second focal distance, and the second focal distance is different from the first focal distance; optimizing the image to be processed using the reference image to eliminate or reduce the influence of the abnormality of the protective cover on the image shot by the first lens. The present disclosure can eliminate or reduce the influence of the abnormality of the protective cover of the lens on the image, thereby improving the image quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, an image processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] As an important carrier of information transmission, images can be applied in more and more scenarios and users have higher and higher requirements for image quality. With the popularization of mobile phones, tablet computers and other terminal devices, people can take pictures anytime and anywhere.

[0003] For terminal equipment, a protective cover is usually provided to protect the lens. Abnormalities of the protective cover will affect the captured image. For example, if the protective cover is dirty, the image quality will be reduced. Summary of the invention

[0004] The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, thereby overcoming the problem of image quality degradation due to abnormal protection cover at least to a certain extent.

[0005] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: obtaining a reference image, the reference image being shot based on a first focusing distance, the first focusing distance being determined according to a distance between a protective cover of a first lens and the first lens; obtaining an image to be processed shot by the first lens based on a second focusing distance, the second focusing distance being different from the first focusing distance; and optimizing the image to be processed using the reference image to eliminate or reduce the influence of an abnormality of the protective cover on the image shot by the first lens.

[0006] According to a second aspect of the present disclosure, an image processing device is provided, including: a first image acquisition module, used to acquire a reference image, the reference image is shot based on a first focusing distance, and the first focusing distance is determined according to the distance between a protective cover of the first lens and the first lens; a second image acquisition module, used to acquire a to-be-processed image shot by the first lens based on a second focusing distance, the second focusing distance being different from the first focusing distance; and an image optimization module, used to optimize the to-be-processed image using the reference image, so as to eliminate or reduce the influence of the abnormality of the protective cover on the image shot by the first lens.

[0007] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned image processing method is implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising a processor; and a memory for storing one or more programs, wherein when the one or more programs are executed by the processor, the processor implements the above-mentioned image processing method.

[0009] In the technical solutions provided in some embodiments of the present disclosure, first, on the one hand, a reference image obtained by shooting based on a first focusing distance is obtained, and the first focusing distance is determined according to the distance between the protective cover of the first lens and the first lens, that is, the reference image is an image obtained by focusing on the protective cover, and on the other hand, an image to be processed shot by the first lens based on a second focusing distance is obtained. Next, the reference image is used to optimize the image to be processed to eliminate or reduce the impact of the abnormality of the protective cover on the image shot by the first lens, thereby improving the quality of the image output by the first lens. The disclosed solution can achieve the effect of improving image quality without adding additional hardware, and the solution has strong universality and can be applied to various types of terminal devices.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 A schematic diagram showing an exemplary architecture of a terminal device according to some embodiments of the present disclosure;

[0013] Figure 2 A schematic diagram showing an exemplary architecture of a terminal device according to some other embodiments of the present disclosure;

[0014] Figure 3 A schematic diagram showing the difference between a normal protective cover plate and an abnormal protective cover plate according to an embodiment of the present disclosure;

[0015] Figure 4 A flowchart schematically shows an image processing method according to an exemplary embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram showing how to obtain a reference image using a first lens in some embodiments of the present disclosure is shown;

[0017] Figure 6 A schematic diagram showing some other embodiments of the present disclosure of using a second lens to obtain a reference image;

[0018] Figure 7 A schematic diagram showing an optimized mode of opening a protective cover abnormality in response to a user operation according to an embodiment of the present disclosure;

[0019] Figure 8 A schematic diagram showing a terminal device according to an embodiment of the present disclosure using a historical image to determine whether to execute the image processing process of the present disclosure;

[0020] Fig. 9 A flowchart schematically showing the process of the image processing method in some embodiments of the present disclosure;

[0021] Fig.10 A flowchart schematically showing the process of the image processing method of some other embodiments of the present disclosure;

[0022] Fig.11 A comparison diagram is shown of an image to be processed before and after optimization using the image processing method according to an embodiment of the present disclosure;

[0023] Fig.12 A block diagram schematically shows an image processing apparatus according to an exemplary embodiment of the present disclosure;

[0024] Fig.13 A block diagram schematically shows an image processing apparatus according to another exemplary embodiment of the present disclosure;

[0025] Fig.14 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0027] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation. In addition, all the terms "first" and "second" below are only for the purpose of distinction and should not be used as limitations of the present disclosure.

[0029] The lens of the terminal device is usually equipped with a protective cover to protect the lens and prevent dust. The material of the protective cover is not limited in the embodiments of the present disclosure, and it can be glass or other materials such as resin. In addition, the protective cover is often transparent. However, the protective cover of the present disclosure can also be a cover that is transparent when shooting and consistent with the color of the terminal device shell at other times. It can be made of a material that becomes transparent after power is turned on, and the present disclosure does not limit this.

[0030] On the one hand, since the protective cover is exposed, the surface of the protective cover is easily dirty, which may come from sweat on the hands, foreign matter contaminated by contact with other objects, dust, rain, etc. Due to the presence of dirt, when shooting with the lens, the corresponding mark of the dirt will appear in the image, resulting in low image quality.

[0031] On the other hand, the terminal device is inevitably bumped during use, which may cause scratches on the protective cover. Similarly, when shooting with the lens, marks corresponding to the scratches will appear in the image, which will also affect the quality of the image.

[0032] In the exemplary embodiment of the present disclosure, dirt, partial shielding, scratches, or even damage can all be considered as abnormalities of the protective cover. It can be understood that the abnormality of the protective cover is relative to the normality of the protective cover which will not affect the captured image.

[0033] In order to at least solve the impact of the abnormal protection cover on the captured image, the embodiment of the present disclosure provides a new image processing solution.

[0034] The image processing scheme of the embodiment of the present disclosure can be implemented by a terminal device, that is, the terminal device can execute each step of the image processing method of the present disclosure, and the following image processing device can be configured in the terminal device. The terminal device may include but is not limited to a smart phone, a tablet computer, a dedicated camera, a laptop computer, a recording instrument, a personal computer, a projector, etc., and the present disclosure does not limit the type of the terminal device.

[0035] However, the image processing solution of the present disclosure can also be implemented by a server, such as the terminal device sends the collected image to the server, and the server performs the image optimization process. The server can also be called a service end, a service platform, a processing platform, etc.

[0036] The following description will be made by taking a terminal device implementing the image processing solution of the embodiment of the present disclosure as an example.

[0037] Figure 1 A schematic diagram showing an exemplary architecture of a terminal device according to some embodiments of the present disclosure is shown. Figure 1 The terminal device 1 at least includes a processor 10, a first lens 11, and a protective cover 12 corresponding to the first lens 11. In an example where the terminal device 1 is a smart phone, the first lens 11 can be a front camera or a rear camera of the smart phone.

[0038] First, on the one hand, a reference image can be obtained by shooting with the first lens 11 based on a first focal distance, where the first focal distance is determined according to the distance between the first lens 11 and the protective cover 12; on the other hand, an image to be processed can also be obtained by shooting with the first lens 11 based on a second focal distance, where the second focal distance is different from the first focal distance. In this case, the first lens 11 can be a zoom camera.

[0039] It should be noted that the embodiment of the present disclosure does not limit the order in which the first lens 11 captures the reference image and the image to be processed. That is, the first lens 11 may first capture the reference image, transmit the reference image to the processor 10, and then capture the image to be processed, and transmit the image to be processed to the processor 10. Alternatively, the first lens 11 may first capture the image to be processed, transmit the image to be processed to the processor 10, and then capture the reference image, and transmit the reference image to the processor 10.

[0040] Next, the processor 10 may optimize the image to be processed by using the reference image to eliminate or reduce the influence of the abnormality of the protective cover on the image captured by the first lens.

[0041] Figure 2 A schematic diagram showing an exemplary architecture of a terminal device according to some other embodiments of the present disclosure is shown. Figure 2, the terminal device 2 at least includes a processor 20, a first lens 21, a second lens 22 and a protective cover 23. Among them, the protective covers of the first lens 21 and the second lens 22 are both protective cover 23, that is, the first lens 21 and the second lens 23 share the same protective cover 23. In the example where the terminal device 2 is a smart phone, the first lens 21 and the second lens 22 can both be front cameras of the smart phone or both be rear cameras of the smart phone.

[0042] First, on the one hand, the reference image can be obtained by shooting with the second lens 22 based on the first focus distance, the first focus distance is determined according to the distance between the first lens 21 and the protective cover 23, and the distance between the first lens 21 and the protective cover 23 is the same as the distance between the second lens 22 and the protective cover 23; on the other hand, the image to be processed can also be obtained by shooting with the first lens 21 based on the second focus distance, and the second focus distance is different from the first focus distance. In this case, the second lens 22 can always focus on the protective cover 23.

[0043] It should be noted that the disclosed embodiment does not limit the order in which the second lens 22 captures the reference image and the first lens 21 captures the image to be processed, and they can also be captured simultaneously. The first lens 21 can transmit the captured image to be processed to the processor 20, and the second lens 22 can transmit the captured reference image to the processor 20.

[0044] Next, the processor 20 may optimize the image to be processed by using the reference image to eliminate or reduce the influence of the abnormality of the protective cover on the image captured by the first lens.

[0045] Figure 3 The difference between normal and abnormal protection cover is shown by example. Figure 3 From the comparison shown, it can be seen that for an abnormal protective cover, the image captured by the corresponding lens will have an imprint that is inconsistent with the real scene.

[0046] The following image processing method can be used to solve the impact of the protective cover abnormality on the quality of the captured image.

[0047] Figure 4 The flowchart of the image processing method according to the exemplary embodiment of the present disclosure is schematically shown. Figure 4 , the image processing method may include the following steps:

[0048] S42. Acquire a reference image, where the reference image is captured based on a first focus distance, and the first focus distance is determined according to a distance between a protective cover of the first lens and the first lens.

[0049] In an exemplary embodiment of the present disclosure, the reference image is photographed based on the first focus distance, and the first focus distance is determined according to the distance between the protective cover of the first lens and the first lens, that is, the reference image is an image obtained by focusing on the protective cover.

[0050] Since dirt or scratches usually appear on the surface of the protective cover plate that is away from the lens (called the outer surface), more specifically, the reference image is an image obtained by focusing on and photographing the outer surface of the protective cover plate.

[0051] According to some embodiments of the present disclosure, the reference image may be obtained by taking a picture with the first lens based on the first focal distance. Figure 5 The first lens 51 can focus and shoot the corresponding protective cover 50 to obtain a reference image.

[0052] Since the first lens needs to capture not only a reference image but also an image to be processed, in this case, the camera may start up slowly during normal shooting.

[0053] In order to solve the problem that the camera may start slowly and affect the user experience, according to other embodiments of the present disclosure, the reference image can be obtained by shooting with the second lens based on the first focus distance. The protective cover of the first lens and the protective cover of the second lens are the same protective cover.

[0054] refer to Figure 6 , the first lens 61 and the second lens 62 can share the same protective cover 60. Since the protective cover 60 is set to be an integrated form for the first lens 61 and the second lens 62, and the dirt or scratches are usually located close to the positions of the two lenses, the second lens 62 can be used to collect the reference image. In other words, the second lens 62 can focus and shoot the protective cover 60 to obtain the reference image.

[0055] S44. Acquire an image to be processed that is photographed by the first lens based on a second focus distance, where the second focus distance is different from the first focus distance.

[0056] The first lens can capture the image to be processed based on the second focal distance. In view of the different types of terminal devices and application scenarios, the second focal distance can be greater than the first focal distance, or the second focal distance can be less than the first focal distance, and the present disclosure does not limit this.

[0057] In some embodiments, the image to be processed is an image obtained by normal shooting, for example, the image to be processed is an image obtained by normal portrait shooting by a smart phone. For another example, the image to be processed is the current frame image when shooting a video. In addition, the image to be processed can also be a preview image.

[0058] It should be understood that the present disclosure does not limit the execution order of step S42 and step S44. That is to say, when implementing the image processing method of the embodiment of the present disclosure, the reference image can be acquired first and then the image to be processed, or the image to be processed can be acquired first and then the reference image.

[0059] S46. Optimize the image to be processed using the reference image to eliminate or reduce the impact of the abnormality of the protective cover on the image captured by the first lens.

[0060] First, the terminal device can use the reference image to determine the image area corresponding to the abnormality of the protective cover from the image to be processed. Specifically, the terminal device can compare the similarity between the reference image and the image to be processed to determine the image area. For example, the image to be processed can be divided into multiple image blocks, and feature point information (such as corner point information) of each image block can be extracted, and the feature point information can be compared with the feature point information of the reference image for similarity, and the image area corresponding to the abnormality of the protective cover can be determined from the image to be processed based on the result of the similarity comparison.

[0061] Next, the terminal device may optimize the image region and merge the optimized image region with the image to be processed to obtain a processed image.

[0062] Specifically, the terminal device can determine the pixel information around the image area in the image to be processed, and estimate the pixel information of the image area using the surrounding pixel information to obtain an optimized image area. For example, the pixel information of the image area can be reassigned by interpolation. The original image area in the image to be processed is replaced with the optimized image area to obtain a processed image.

[0063] In an embodiment in which the image area corresponding to the protective cover in the image to be processed is the area after the image is blurred, for example, when capturing a portrait, the image area to be processed except for the portrait is blurred. In this case, the degree of blurring of the image area can be used to optimize the image area. Specifically, the degree of blurring can be characterized by the diameter of the circle of confusion, and the diameter of the circle of confusion can be determined based on the object distance, the focus distance, the focal length and the aperture value. When the degree of blurring is obtained, a pixel-level brightness subtraction operation can be performed based on the degree of blurring to obtain a processed image.

[0064] In addition, it is understandable that, when the protective cover is normal, in order to save resources of the terminal device, it is not necessary to execute the above-mentioned image processing method of the present disclosure. That is, before executing the above-mentioned image processing method of steps S42 to S46, it is also possible to determine whether the protective cover is normal.

[0065] According to some embodiments of the present disclosure, the user can determine whether the protective cover is abnormal. If the human eye finds that the protective cover is abnormal, the above method process can be started by manual setting.

[0066] Specifically, the terminal device may respond to the start-up operation of the optimization mode for the abnormality of the protective cover, and the optimization mode for the abnormality of the protective cover is started.

[0067] like Figure 7 As shown, the camera setting interface 72 can be entered in the shooting preview interface 71 to check the start button of the abnormal optimization of the protective cover. Figure 7 In addition to the scenario shown, a shortcut button for an optimization mode for abnormal protection cover can also be set in the shooting preview interface 71 to directly turn on the optimization mode in the shooting preview interface 71 .

[0068] When the optimization mode for the abnormality of the protective cover is turned on, the terminal device can respond to the shooting operation for the first lens, such as the terminal device responds to the user clicking the shooting button, obtains the reference image and the image to be processed, and uses the reference image to optimize the image to be processed, that is, executes the above-mentioned process of steps S42 to S46.

[0069] According to some other embodiments of the present disclosure, the terminal device can determine by itself whether to execute the above-mentioned image processing method of the present disclosure in combination with multiple historical images.

[0070] Specifically, the terminal device may respond to the shooting operation for the first lens and obtain multiple historical images shot by the first lens, such as 3 historical images, 5 historical images, etc. When it is determined that the protective cover of the first lens is abnormal according to the multiple historical images, the terminal device may obtain a reference image and an image to be processed, and optimize the image to be processed using the reference image, that is, perform the process from step S42 to step S46 above.

[0071] For example, the terminal device may compare the similarity of multiple historical images. If the historical images are shot in different scenes but have image areas with similar positions and contents, the protective cover is considered to be abnormal.

[0072] refer to Figure 8 The terminal device can use multiple historical images to determine whether the protective cover is normal or abnormal. When the protective cover is normal, the terminal device can perform conventional shooting processing; when the protective cover is abnormal, the terminal device can apply the above-mentioned image processing process of the present disclosure.

[0073] refer to Fig. 9 , the process of the image processing method of some embodiments of the present disclosure is explained.

[0074] In step S902, the terminal device uses the first lens to focus and shoot the protective cover to obtain a reference image;

[0075] In step S904, the terminal device responds to the user's shooting operation and uses the first lens to shoot normally to obtain an image to be processed;

[0076] In step S906, the terminal device optimizes the image to be processed using the reference image to obtain an optimized image;

[0077] In step S908, the terminal device may output the optimized image. Outputting the optimized image may mean directly saving the optimized image, uploading it to the cloud or other devices, or using the optimized image as input to further perform subsequent image processing, which is not limited in the present disclosure.

[0078] refer to Fig.10 , the process of the image processing method of other embodiments of the present disclosure is explained.

[0079] In step S1002, the terminal device responds to the user's shooting operation and uses the first lens to shoot normally to obtain an image to be processed;

[0080] In step S1004, the terminal device uses the second lens to focus and shoot the protective cover to obtain a reference image;

[0081] In step S1006, the terminal device optimizes the image to be processed using the reference image to obtain an optimized image;

[0082] In step S1008, the terminal device may output the optimized image. Outputting the optimized image may mean directly saving the optimized image, uploading it to the cloud or other devices, or using the optimized image as input to further perform subsequent image processing, which is not limited in the present disclosure.

[0083] Fig.11 The figure shows a comparison of the image to be processed before and after optimization using the image processing method of the embodiment of the present disclosure. Fig.11 After the image to be processed undergoes the image processing process of the exemplary embodiment of the present disclosure, the influence of the abnormality of the protective cover on the captured image can be eliminated, thereby improving the quality of the image.

[0084] It should be understood that even if the above image processing method cannot completely eliminate the impact of the abnormality of the protective cover on the shooting result, it can reduce the impact of the abnormality of the protective cover on the captured image and can also improve the image quality to a certain extent.

[0085] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0086] Furthermore, this exemplary embodiment also provides an image processing device.

[0087] Fig.12 FIG. 1 schematically shows a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure. Fig.12 , the image processing device 12 according to an exemplary embodiment of the present disclosure may include a first image acquisition module 121 , a second image acquisition module 123 , and an image optimization module 125 .

[0088] Specifically, the first image acquisition module 121 can be used to acquire a reference image, which is taken based on a first focusing distance, and the first focusing distance is determined according to the distance between the protective cover of the first lens and the first lens; the second image acquisition module 123 can be used to acquire a to-be-processed image taken by the first lens based on a second focusing distance, and the second focusing distance is different from the first focusing distance; the image optimization module 125 can be used to optimize the to-be-processed image using the reference image to eliminate or reduce the impact of the abnormality of the protective cover on the image taken by the first lens.

[0089] According to an exemplary embodiment of the present disclosure, the first image acquisition module 121 may be configured to execute: acquiring a reference image captured by a first lens based on a first focus distance.

[0090] According to an exemplary embodiment of the present disclosure, the first image acquisition module 121 may also be configured to execute: acquiring a reference image taken by the second lens based on the first focus distance; wherein the protective cover of the first lens and the protective cover of the second lens are the same protective cover.

[0091] According to an exemplary embodiment of the present disclosure, referring to Fig.13 Compared with the image processing device 12 , the image processing device 13 may further include a processing start module 131 .

[0092] Specifically, the processing start module 131 can be configured to execute: responding to the start operation of the optimization mode for the protective cover abnormality, the optimization mode for the protective cover abnormality is turned on; responding to the shooting operation for the first lens, obtaining a reference image and an image to be processed, and using the reference image to optimize the image to be processed.

[0093] According to an exemplary embodiment of the present disclosure, the processing start module 131 can be configured to perform: in response to a shooting operation for a first lens, obtaining multiple historical images shot by the first lens; when it is determined based on the multiple historical images that the protective cover of the first lens is abnormal, obtaining a reference image and an image to be processed, and using the reference image to optimize the image to be processed.

[0094] According to an exemplary embodiment of the present disclosure, the image optimization module 125 can be configured to perform: using a reference image, determining an image area corresponding to the protective cover abnormality from the image to be processed; optimizing the image area, and merging the optimized image area with the image to be processed to obtain a processed image.

[0095] According to an exemplary embodiment of the present disclosure, the process of optimizing the image area by the image optimization module 125 can be configured to perform: determining pixel information around the image area; using the pixel information around the image area, estimating the pixel information of the image area to obtain an optimized image area.

[0096] Since the functional modules of the image processing device in the embodiment of the present disclosure are the same as those in the above-mentioned method embodiment, they will not be described in detail here.

[0097] Fig.14 Schematic diagram of an electronic device suitable for implementing the exemplary embodiment of the present disclosure is shown. The terminal device of the exemplary embodiment of the present disclosure may be configured as follows Fig.14 It should be noted that Fig.14 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0098] The electronic device of the present disclosure includes at least a processor and a memory, wherein the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the image processing method of the exemplary embodiment of the present disclosure.

[0099] Specifically, Fig.14As shown, the electronic device 140 may include: a processor 1410, an internal memory 1421, an external memory interface 1422, a Universal Serial Bus (USB) interface 1430, a charging management module 1440, a power management module 1441, a battery 1442, an antenna 1, an antenna 2, a mobile communication module 1450, a wireless communication module 1460, an audio module 1470, a sensor module 1480, a display screen 1490, a camera module 1491, an indicator 1492, a motor 1493, a button 1494, and a Subscriber Identification Module (SIM) card interface 1495, etc. The sensor module 1480 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0100] It is to be understood that the structure illustrated in the embodiment of the present disclosure does not constitute a specific limitation on the electronic device 140. In other embodiments of the present disclosure, the electronic device 140 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0101] The processor 1410 may include one or more processing units, for example, the processor 1410 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 1410 for storing instructions and data.

[0102] The electronic device 140 can realize the shooting function through the ISP, the camera module 1491, the video codec, the GPU, the display screen 1490 and the application processor. In some embodiments, the electronic device 140 may include 1 or N camera modules 1491, where N is a positive integer greater than 1. If the electronic device 140 includes N lenses, one of the N lenses is a main lens.

[0103] The internal memory 1421 may be used to store computer executable program codes, which may include instructions. The internal memory 1421 may include a program storage area and a data storage area. The external memory interface 1422 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 140.

[0104] The present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.

[0105] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0106] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0107] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the embodiments of the present disclosure.

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0110] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0111] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0112] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0113] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0114] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, It is characterized in that include: Acquire a reference image, where the reference image is captured based on a first focus distance, where the first focus distance is determined based on a distance between a protective cover of a first lens and the first lens; Acquire an image to be processed that is taken by the first lens based on a second focusing distance, where the second focusing distance is different from the first focusing distance; Using the reference image, determining an image region corresponding to the abnormality of the protective cover from the image to be processed, optimizing the image region, and fusing the optimized image region with the image to be processed to obtain a processed image, so as to eliminate or reduce the influence of the abnormality of the protective cover on the image captured by the first lens; Wherein, obtaining the reference image includes: Acquire a reference image captured by a second lens based on the first focus distance, wherein the protective cover of the first lens and the protective cover of the second lens are the same protective cover; The image processing method further comprises: In response to the start-up operation of the optimization mode for the abnormality of the protective cover, the optimization mode for the abnormality of the protective cover is turned on; in response to the shooting operation for the first lens, the reference image and the image to be processed are acquired, and the image to be processed is optimized using the reference image; or In response to a shooting operation for a first lens, a plurality of historical images shot by the first lens are obtained; when it is determined that the protective cover of the first lens is abnormal based on the plurality of historical images, the reference image and the image to be processed are obtained, and the image to be processed is optimized using the reference image.

2. The image processing method according to claim 1, It is characterized in that Optimizing the image area includes: Determining pixel information around the image area; The pixel information of the image region is estimated by using the pixel information around the image region to obtain an optimized image region.

3. An image processing device, It is characterized in that include: A first image acquisition module, configured to acquire a reference image, wherein the reference image is captured based on a first focus distance, wherein the first focus distance is determined according to a distance between a protective cover of a first lens and the first lens; A second image acquisition module, used to acquire an image to be processed shot by the first lens based on a second focus distance, where the second focus distance is different from the first focus distance; an image optimization module, configured to use the reference image to determine an image region corresponding to an abnormality of the protective cover from the image to be processed, optimize the image region, and fuse the optimized image region with the image to be processed to obtain a processed image, so as to eliminate or reduce the influence of the abnormality of the protective cover on the image captured by the first lens; A processing start-up module, configured to respond to a start-up operation of the optimization mode for the abnormal protection cover, and start the optimization mode for the abnormal protection cover; respond to a shooting operation for the first lens, obtain the reference image and the image to be processed, and optimize the image to be processed by using the reference image; or respond to a shooting operation for the first lens, obtain multiple historical images shot by the first lens; when it is determined that the protective cover of the first lens is abnormal according to the multiple historical images, obtain the reference image and the image to be processed, and optimize the image to be processed by using the reference image; The first image acquisition module is configured to acquire a reference image captured by the second lens based on the first focusing distance, and the protective cover of the first lens and the protective cover of the second lens are the same protective cover.

4. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the image processing method according to claim 1 or 2 is implemented.

5. An electronic device, It is characterized in that include: processor; A memory for storing one or more programs, which, when executed by the processor, enables the processor to implement the image processing method as claimed in claim 1 or 2.