Image processing method and apparatus, computer readable medium, and electronic device

Abstract

The present disclosure provides an image processing method and device, a computer readable medium and an electronic device, and relates to the technical field of image shooting. The method comprises: acquiring a shooting image, wherein the shooting image comprises a first image and a second image obtained by different cameras shooting a same shooting target; performing occlusion detection based on the first image and the second image to obtain an occlusion detection result; and acquiring a corresponding depth image of at least one of the first image and the second image based on the occlusion detection result. The present disclosure can select a suitable image for depth estimation according to the occlusion detection result in the case that the camera exists occlusion, improve the accuracy of the depth image, and further ensure the image quality of the output image, such as the image blurring effect of the output image.

Description

Technical Field

[0001] This disclosure relates to the field of image capture technology, and more specifically to an image processing method, an image processing apparatus, a computer-readable medium, and an electronic device. Background Technology

[0002] With the continuous improvement of people's living standards, imaging has become a core function of mobile phone products. In order to improve the imaging effect, it is necessary to determine the depth information of each pixel in the image in some scenarios (such as image blurring scenarios). However, hardware-based depth detection has high hardware costs and a wide range of applications. Therefore, the imaging field has proposed a solution to implement depth estimation in software, namely monocular depth estimation and binocular depth estimation.

[0003] Currently, in relevant binocular depth estimation schemes, if the camera is obstructed, the depth estimation results may be inaccurate. Summary of the Invention

[0004] The purpose of this disclosure is to provide an image processing method, image processing apparatus, computer-readable medium, and electronic device that can effectively improve the accuracy of depth estimation results when the camera is occluded.

[0005] According to a first aspect of this disclosure, an image processing method is provided, comprising:

[0006] Acquire captured images, including a first image and a second image obtained by capturing the same target with different cameras;

[0007] Occlusion detection is performed based on the first image and the second image to obtain occlusion detection results;

[0008] Based on the occlusion detection results, at least one corresponding depth image of the first image and the second image is obtained.

[0009] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising:

[0010] The image acquisition module is used to acquire captured images, including a first image and a second image obtained by capturing the same target with different cameras;

[0011] An occlusion detection module is used to perform occlusion detection based on the first image and the second image, and obtain occlusion detection results;

[0012] The depth estimation module is used to obtain at least one corresponding depth image of the first image and the second image based on the occlusion detection result.

[0013] According to a third aspect of this disclosure, a computer-readable medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method described above.

[0014] According to a fourth aspect of this disclosure, an electronic device is provided, characterized in that it comprises:

[0015] Processor; and

[0016] Memory is used to store one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the methods described above.

[0017] One embodiment of this disclosure provides an image processing method that can acquire captured images, namely a first image and a second image obtained by different cameras capturing the same target. Occlusion detection can be performed based on the first and second images to determine the occlusion detection result. Then, at least one of the first and second images can be selected for depth estimation based on the occlusion detection result to determine the depth image corresponding to the captured image. By selecting a suitable image as input for depth estimation based on the occlusion detection result, the accuracy of the depth image can be effectively improved even when the camera is occluded, thereby ensuring the image quality of the output image, such as improving the image blurring effect of the output image.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0020] Figure 1 A schematic diagram illustrates an exemplary application environment in which an image processing method and apparatus according to embodiments of the present disclosure can be applied;

[0021] Figure 2 The illustration schematically shows a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;

[0022] Figure 3 The illustration schematically shows a process diagram for determining a depth image in an exemplary embodiment of the present disclosure;

[0023] Figure 4This schematically illustrates a process diagram for depth image determination under different occlusion areas in an exemplary embodiment of the present disclosure;

[0024] Figure 5 This illustration schematically shows a flowchart of determining an occlusion detection result in an exemplary embodiment of the present disclosure;

[0025] Figure 6 This illustration schematically shows a principle diagram of implementing occlusion detection of a first image and a second image in an exemplary embodiment of the present disclosure;

[0026] Figure 7 This schematic diagram illustrates a process for image blurring of captured images according to an exemplary embodiment of the present disclosure.

[0027] Figure 8 This diagram illustrates an example of displaying an occlusion warning message in an exemplary embodiment of the present disclosure.

[0028] Figure 9 This schematic diagram illustrates the composition of an image processing apparatus in an exemplary embodiment of the present disclosure.

[0029] Figure 10 A schematic diagram of an electronic device to which embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0030] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0031] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] Figure 1 A schematic diagram of a system architecture for an exemplary application environment in which an image processing method and apparatus according to embodiments of the present disclosure can be applied is shown.

[0033] refer to Figure 1As shown, in a possible application scenario, the image processing method in this embodiment can be applied to terminal device 110, and correspondingly, an image processing device can also be installed in terminal device 110. Specifically, terminal device 110 may have a rear camera module, and capture images through the rear camera module. The rear camera module may include a main camera 111 and a secondary camera 112, thereby obtaining a first image through the main camera 111 and a second image through the secondary camera 112.

[0034] The main camera 111 can be considered equivalent to a wide-angle lens with a focal length of approximately 28mm. This focal length is close to the field of view of the human eye, meaning what you see is what you capture. It is the most frequently used camera on widely used terminal devices. When you open the phone's camera app, you can usually see a 1X value displayed. The focal length corresponding to 1X is the main camera's focal length, which offers the best image quality among all focal lengths and is suitable for shooting portraits, architecture, landscapes, and documentary subjects.

[0035] The secondary camera 112 refers to a camera used to assist or supplement the focal length or deficiencies of the main camera 111. For example, the secondary camera 112 can be an ultra-wide-angle camera. Compared with the main camera 111, the ultra-wide-angle camera has a larger field of view and can capture a wider scene from the same position, making it suitable for shooting landscapes and architecture. The ultra-wide-angle camera corresponds to the focal length of 0.5X-0.6X in mobile phone cameras. The secondary camera 112 can also be a telephoto camera; generally, focal lengths above 1X are usually referred to as telephoto. The higher the coefficient before X in telephoto, the farther the image can be captured. Telephoto lenses can take higher quality photos from a distance, capturing distant objects or magnifying objects in the scene without degrading image quality like digital zoom. Telephoto lenses can "bring closer" the distance between the background and foreground, creating a sense of spatial compression and making the overall image fuller. This "compression" is one of the characteristics of telephoto lenses. Telephoto lenses have less distortion and weaker perspective effects, which can bring the foreground and background closer, enhancing the relationship between them and creating unique visual effects. Of course, the secondary camera can also be other types of cameras, including but not limited to infrared cameras, depth cameras, etc. This example embodiment does not make any special limitations on this. The secondary camera 112 can be one of the above-mentioned different types of cameras, or a combination of multiple types. This example embodiment is not limited to this.

[0036] In one possible application scenario, the image processing method in this embodiment can be applied to terminal device 120, and correspondingly, an image processing device can also be disposed in terminal device 120. Specifically, terminal device 120 may have a front-facing camera module, and capture images through the front-facing camera module. The front-facing camera module may include at least a main camera 121 and a secondary camera 122, so that the main camera 121 can capture a first image and the secondary camera 122 can capture a second image. It is understood that the type and related settings of the main camera 121 and the secondary camera 122 are the same as those of the main camera 111 and the secondary camera 112 of terminal device 110, and will not be described again here.

[0037] In one possible application scenario, the image processing method in this embodiment can be applied to the camera system 130. Correspondingly, the image processing device can also be installed in the terminal device of the camera system 130, and the camera system 130 can capture images. Specifically, the camera system 130 may include a terminal device with only one camera 131 and an external camera 132 or other terminal device 133 connected to the terminal device via a network (such as a wired network, wireless network, etc.). The camera 131 can be used as the main camera of the camera system 130, and the external camera 132 or other terminal device 133 can be used as the secondary camera of the camera system 130. Thus, a first image can be captured through the camera 131, and a second image can be captured through the external camera 132 or other terminal device 133. Alternatively, the external camera 132 can be used as the main camera of the camera system 130, and the camera 131 can be used as the secondary camera of the camera system 130. This example embodiment does not impose any special limitations on this.

[0038] In one possible application scenario, the image processing method can also be executed by a server connected to the terminal device 110, terminal device 120, or camera system 130 via a network. Correspondingly, the image processing device can also be located in the server. For example, in an exemplary embodiment, the terminal device 110, terminal device 120, or camera system 130 may acquire a first image and a second image, and upload the first and second images to the server. After the server generates a target image using the image processing method provided in this embodiment, it transmits the target image to the terminal device 110, terminal device 120, or camera system 130, etc.

[0039] In related technologies, traditional dual-camera image blurring solutions utilize dual / multiple cameras for depth calculation. When cameras are obstructed or their image sensors are partially damaged, depth estimation becomes impossible, or the estimated depth fails to achieve image blurring, thus reducing the user experience during daily shooting. Specifically, in some scenarios, the obstruction during shooting is weak, but some solutions lack obstruction detection or fail to detect even weak obstructions, resulting in low accuracy of depth estimation and consequently affecting the image quality of the output image, such as the blurred image.

[0040] Based on one or more problems in the related technologies, this disclosure first provides an image processing method. The following describes the image processing method and image processing apparatus of the exemplary embodiments of this disclosure by taking the terminal device 110, terminal device 120 or the terminal device in the camera system 130 as examples of the method being executed.

[0041] Figure 2 A flowchart illustrating an image processing method according to this exemplary embodiment is shown, which may include the following steps S210 to S240:

[0042] In step S210, captured images are obtained, including a first image and a second image obtained by different cameras capturing the same target.

[0043] In an exemplary embodiment, capturing an image refers to at least one frame of an image captured by a camera activated when the user triggers a shooting command. Specifically, capturing an image may include a first image and a second image obtained by different cameras capturing the same target. The different cameras may be at least two cameras activated by the terminal device. The at least two cameras may be a combination of the main camera and a secondary camera of the terminal device, or a combination of secondary cameras of different types. For example, the different cameras may be a combination of the main camera and an ultra-wide-angle camera, or a combination of the main camera and a telephoto camera. Of course, it may also be a combination of an ultra-wide-angle camera and a telephoto camera. This example embodiment is not limited to this.

[0044] Understandably, the first image can be an image captured by the main camera, i.e., a preview image displayed on the graphical user interface of the terminal device, which is visible to the user; the second image can be an image captured by the secondary camera. Since at least two cameras are activated simultaneously during shooting and two images are captured, the graphical user interface of the terminal device can generally only display one image. Therefore, the second image may not be visible to the user. That is, the captured second image may be image data used only to assist the first image in depth estimation and image blurring. Of course, in some scenarios, both the first image and the second image can be displayed in the graphical user interface. This example embodiment does not impose any special limitations on this.

[0045] It is understandable that different cameras can also be cameras with the same shooting parameters, without needing to distinguish between a main camera and a secondary camera. In this case, the resolution and clarity of both the first and second images can meet the display requirements. Therefore, the first image can be used as the preview image and the second image as the auxiliary image, or vice versa. This example embodiment is not limited to this. For ease of understanding and explanation, the following description will be based on the case where the first image is the preview image and the second image is the auxiliary image.

[0046] It should be noted that the terms "first image" and "second image" in the embodiments of this disclosure are only used to distinguish images captured by different cameras, and have no special meaning, and should not impose any special limitations on this example embodiment.

[0047] In step S220, occlusion detection is performed based on the first image and the second image to obtain the occlusion detection result.

[0048] In one exemplary embodiment, occlusion detection refers to a processing method that compares and analyzes the image content of a first image and the image content of a second image to determine whether the first or second image contains occlusions or abnormal regions. For example, occlusion detection can be a processing method that determines the statistical histograms of the first and second images and analyzes them to determine whether the first or second image contains occlusions or abnormal regions; it can also be a processing method that determines the image hash results of the first and second images and analyzes them to determine whether the first or second image contains occlusions or abnormal regions. Of course, other processing methods that can analyze image similarity or image occlusion can also be used. A deep learning model for image occlusion detection can be pre-trained, and image occlusion detection can be performed based on the deep learning model. This example embodiment is not limited to this.

[0049] Occlusion detection results refer to the data obtained after performing occlusion detection on the first image and the second image. For example, occlusion detection results may include whether there is occlusion in the image and the corresponding occlusion area when occlusion exists. Of course, occlusion detection results may also be data such as the location coordinates of the occlusion area. This example embodiment does not make any special limitations on this.

[0050] For example, the image hash results of the first image and the second image can be calculated, and the image similarity can be calculated based on the hash results of the two images. When the image similarity is greater than or equal to a certain threshold, it can be considered that the first image or the second image has an occlusion. The occlusion area is determined based on the image similarity, and the existence of occlusion in the first image or the second image, as well as the occlusion area when occlusion exists, is taken as the occlusion detection result.

[0051] Understandably, this embodiment generally considers the case where only one of the two cameras is obstructed, or the case where at least one of the two cameras that is not used for preview image output is obstructed. For example, the first image and the second image can be obtained by capturing the main camera and the secondary camera of the terminal device. In this case, since the second image captured by the secondary camera is not visible to the user, the user is unaware of the obstruction when the secondary camera (or the second image) is obstructed, so obstruction detection is required. However, for the case where both cameras are obstructed, since at least one image will be presented to the user, the user can perceive it and can avoid or deal with the obstruction at any time, so obstruction detection is not required.

[0052] In step S230, based on the occlusion detection result, at least one corresponding depth image of the first image and the second image is obtained.

[0053] In an exemplary embodiment, depth estimation can be performed by selecting at least one of the first image and the second image based on the occlusion detection result. For example, if the occlusion detection result indicates that the second image is not occluded, then a stereo matching can be directly performed based on the first image and the second image to obtain the depth image. If the occlusion detection result indicates that the second image is occluded and the occlusion area is large, then the second image can be disregarded to avoid introducing large errors, and the monocular depth estimation result of the first image can be directly used as the depth image. If the occlusion detection result indicates that the second image is occluded and the occlusion area is small, then the second image can be introduced to improve the accuracy of the depth estimation result. Therefore, the monocular depth estimation result of the first image and the stereo depth estimation result of the first image and the stereo depth estimation result of the second image can be directly fused to obtain the depth image, thereby improving the accuracy of the depth image.

[0054] Optionally, in an exemplary embodiment, a target image can be determined based on the depth image and the captured image, and the target image can be used as the shooting output. The target image refers to an image obtained by combining the depth image with the captured image and performing appropriate processing. For example, the target image can be an image with a blurred effect obtained by distinguishing the shooting target (such as a human figure) and the background area in the captured image using the depth image and blurring the background area; or it can be an image with a brightness enhancement effect obtained by adjusting the brightness of different areas in the captured image using the depth value in the depth image. Of course, the target image can also be other images that can be obtained by adjusting the captured image based on the depth image; this example embodiment does not specifically limit this.

[0055] Taking the case where the first image is a preview image and the second image is an auxiliary image as an example, the first image can be adjusted based on the depth image to determine the target image; taking the case where the second image is a preview image and the first image is an auxiliary image as an example, the second image can also be adjusted based on the depth image to determine the target image; of course, the image with the highest clarity in the first or second image, or the fused image of the two images, can also be adjusted based on the depth image to obtain the target image. This example embodiment does not impose any special limitations on this.

[0056] By selecting an appropriate image as input through occlusion detection results for depth estimation, the accuracy of the depth image can be effectively improved even when the camera is occluded, thereby ensuring the image quality of the output target image, such as improving the image blurring effect of the target image.

[0057] The technical content of steps S210 to S240 will be described in detail below.

[0058] In an exemplary embodiment, taking the case where the first image is a preview image and the second image is an auxiliary image as an example, the occlusion detection result may include the presence of occlusion in the second image and the occlusion area when the second image is occluded.

[0059] Optionally, if the occlusion detection result indicates that the second image is occluded and the occlusion area is greater than or equal to the target threshold, then the first image is selected; and monocular depth estimation is performed based on the first image to determine the depth image corresponding to the captured image.

[0060] The target threshold refers to a pre-set data set to determine the degree of influence of the occlusion region on the depth estimation result. For example, the target threshold could be 75%. If the occlusion area of ​​the second image is detected to be greater than or equal to 75%, the occlusion area of ​​the second image can be considered too large and cannot be used as input for the depth estimation result. Therefore, the first image can be selected for depth estimation. Specifically, monocular depth estimation is performed based on the first image. For example, the first image can be input into a pre-trained depth estimation model to determine the depth image corresponding to the captured image. Of course, traditional cue-based methods (such as motion estimation, linear perspective, shadow cues, focus cues, etc.) can also be used to perform monocular depth estimation on the first image to determine the depth image corresponding to the captured image. This example embodiment does not impose any special limitations on the monocular depth estimation method. Of course, the target threshold can also be 50%, etc., and can be customized according to the actual application situation. This embodiment is not limited to this.

[0061] When the occlusion area of ​​the second image is large, the first image can be selected for monocular depth estimation to obtain a depth image. This can effectively avoid the introduction of large errors in the binocular depth estimation process due to the large occlusion area of ​​the second image, thus ensuring the accuracy of the depth image.

[0062] Optionally, when the occlusion area is smaller than the target threshold, it can be achieved by... Figure 3 The steps in the document are used to perform depth estimation, refer to... Figure 3 As shown, it can specifically include:

[0063] Step S310: When the occlusion area of ​​the second image is less than the target threshold, then select both the first image and the second image; and

[0064] Step S320: Perform monocular depth estimation based on the first image to determine the first depth estimation result;

[0065] Step S330: Perform binocular depth estimation based on the first image and the second image to determine the second depth estimation result;

[0066] Step S340: According to the preset first fusion weight, the first depth estimation result and the second depth estimation result are fused to obtain the depth image corresponding to the captured image.

[0067] The first depth estimation result refers to the depth image obtained by performing monocular depth estimation on the first image, and the second depth estimation result refers to the depth image obtained by performing binocular depth estimation on the first image and the second image.

[0068] When the occlusion detection result indicates that the second image is occluded and the occlusion area is less than the target threshold, it can be considered that the occlusion area in the second image is small, and the error introduced by the occlusion area of ​​the second image is within an acceptable range. At this time, the first image and the second image can be selected as the input for depth estimation.

[0069] Specifically, the first fusion weight refers to the pre-set weight data used to fuse the first depth estimation result and the second depth estimation result. For example, since the binocular depth estimation result is more accurate than the monocular depth estimation result, the second depth estimation result can be used as the primary depth estimation result. In this case, the first fusion weight can be 0.3 for the first depth estimation result and 0.7 for the second depth estimation result; or it can be 0.4 for the first depth estimation result and 0.6 for the second depth estimation result. Of course, if the occlusion area is still large, the monocular depth estimation result is more accurate than the binocular depth estimation result, and the first depth estimation result can also be used as the primary depth estimation result. That is, the weight of the first depth estimation result in the first fusion weight is greater than the weight of the second depth estimation result. The setting of the first fusion weight can be customized according to the actual situation. For example, multiple levels of the first fusion weight can be set according to the occlusion area of ​​the second image. This example embodiment is not limited to this.

[0070] Monocular depth estimation can be performed based on the first image to determine the first depth estimation result. Binocular depth estimation can be performed based on the first image and the second image to determine the second depth estimation result. The first depth estimation result and the second depth estimation result are fused together using a first fusion weight to obtain a depth image. The obtained depth image has higher accuracy than the first depth estimation result alone. Compared with the second depth estimation result alone, it can compensate for the error introduced by the occluded area, effectively improving the accuracy of the depth image.

[0071] Figure 4 The illustration schematically depicts a process diagram for depth image determination under different occlusion areas in an exemplary embodiment of the present disclosure.

[0072] refer to Figure 4As shown, when the occlusion detection result indicates that the second image is occluded, the occlusion area 401 of the second image can be obtained. If the occlusion area 401 of the second image is greater than or equal to the target threshold, the first image can be selected, and monocular depth estimation is performed based on the first image. The monocular depth estimation result 402 is used as the depth image 403 corresponding to the captured image. If the occlusion area 401 of the second image is less than the target threshold, the first image and the second image can be selected, and monocular depth estimation is performed based on the first image to determine the monocular depth estimation result 402. Binocular stereo matching is then performed based on the first image and the second image to obtain the binocular stereo matching result 404. The monocular depth estimation result 402 and the binocular stereo matching result 404 are then weighted and fused using a preset first fusion weight to obtain a weighted fusion result 405. The weighted fusion result 405 is used as the depth image 403 corresponding to the captured image.

[0073] In an exemplary embodiment, the occlusion detection result may include the second image not being occluded. Specifically, if the occlusion detection result is determined to be that the second image is not occluded, then the first image and the second image are selected; binocular depth estimation is performed based on the first image and the second image to determine the depth image corresponding to the captured image.

[0074] When no occlusion is detected in the second image, the binocular depth estimates of the first and second images are directly used as the depth image. Compared with monocular depth estimation, the obtained depth image is more accurate, ensuring the accuracy of the depth estimation results.

[0075] In an exemplary embodiment, occlusion detection can be performed on the first image and the second image in the following manner: the statistical histograms corresponding to the first image and the second image can be determined, and occlusion detection can be performed on the first image and the second image based on the statistical histograms to determine the occlusion detection result.

[0076] Among them, the statistical histogram refers to the histogram curve obtained by statistically analyzing the pixel data in the image. The statistical histogram can effectively show the distribution of each feasible pixel in the image. When the image content is not much different, the statistical histogram can be used to quickly and efficiently analyze the differences between two images and realize occlusion detection.

[0077] Optionally, statistical histograms may include grayscale histograms and gradient histograms. A grayscale histogram is a histogram obtained by statistically analyzing the grayscale values ​​of an image, and it is a distribution curve from 0 to 255. A gradient histogram (Histogram of Oriented Gradient, HOG) is a histogram obtained by statistically analyzing the gradient and orientation of an image, and it is a distribution curve from 0 to 255.

[0078] It can be done Figure 5 The steps in the document implement occlusion detection for the first and second images, referencing... Figure 5 As shown, it can specifically include:

[0079] Step S510: Determine the first correlation coefficient based on the grayscale histograms corresponding to the first image and the second image respectively;

[0080] Step S520: Determine the second correlation coefficient based on the gradient histograms corresponding to the first image and the second image respectively;

[0081] Step S530: The first correlation coefficient and the second correlation coefficient are fused according to the preset second fusion weight, and the occlusion detection result is determined based on the fusion result.

[0082] The first correlation coefficient refers to the correlation coefficient between the first image and the second image obtained by performing correlation operations on the gray-level histograms corresponding to the first image and the second image respectively. For example, each element in the gray-level histogram (or the vector converted from it) of the two images can be multiplied, and the sum of the products can be used as the first correlation coefficient. Of course, the gray-level histogram can also be converted in other ways to obtain the first correlation coefficient (a probability value). This example embodiment is not limited to this.

[0083] The second correlation coefficient refers to the correlation coefficient between the first image and the second image obtained by performing correlation operations on the gradient histograms corresponding to the first image and the second image respectively. For example, each element in the gradient histogram (or the vector converted) of the two images can be multiplied, and the sum of the products can be used as the second correlation coefficient. Of course, the gradient histogram can also be converted in other ways to obtain the second correlation coefficient (a probability value). This example embodiment is not limited to this.

[0084] The second fusion weight refers to the pre-set weight data used to fuse the first correlation coefficient and the second correlation coefficient. For example, the second fusion weight can be 0.5 for the first correlation coefficient and 0.5 for the second correlation coefficient. Of course, it can also be 0.3 and 0.7, etc. The second fusion weight can be customized according to the actual application situation. This embodiment does not make any special limitations on this.

[0085] The first correlation coefficient and the second correlation coefficient can be fused according to the preset second fusion weight, and the occlusion detection result can be determined based on the fusion result (a probability value). For example, if the fusion result is greater than a certain value, it can be considered that there is an occlusion area in the second image, and the occlusion area can be calculated based on the fusion result. If the fusion result is a probability value, the probability value can be used as the occlusion area.

[0086] Figure 6 The illustration schematically shows a principle diagram of implementing occlusion detection of a first image and a second image in an exemplary embodiment of the present disclosure.

[0087] refer to Figure 6 As shown, a first image 601 and a second image 602 can be acquired. A first correlation coefficient 603 is determined based on the grayscale histograms corresponding to each of the first and second images 601 and 602, and a second correlation coefficient 604 is determined based on the gradient histograms corresponding to each of the first and second images 601 and 602. A preset second fusion weight can be acquired, and the first and second correlation coefficients are weighted and fused according to the preset second fusion weight to obtain a weighted fusion result 605. The occlusion detection result is then determined based on the weighted fusion result 605.

[0088] By calculating the grayscale histogram and gradient histogram between the first and second images, and fusing these two data points to obtain the occlusion detection result, the accuracy of the occlusion detection result can be effectively improved.

[0089] In an exemplary embodiment, occlusion detection can be performed on a first image and a second image by: determining a first image hash result for the first image and a second image hash result for the second image; calculating the Hamming distance between the first image hash result and the second image hash result, and determining image similarity based on the Hamming distance; and then determining the occlusion detection result based on the image similarity.

[0090] The image hash results of the first image and the second image can be calculated, and then the image similarity between the two images can be calculated through the image hash results. When the image similarity is greater than a certain value, it can be considered that there is occlusion detection in the second image, and the occlusion area can be determined according to the image similarity. For example, if the image similarity is 60%, then the occlusion area can be 40% (100%-60%).

[0091] By using the image hashing results, the image similarity between the first and second images can be quickly calculated, and the occlusion detection result can be determined based on the image similarity, which effectively improves the efficiency of occlusion detection and ensures the accuracy of the occlusion detection result.

[0092] In an exemplary embodiment, a target image region corresponding to the target being captured can be determined in the captured image based on the depth image, and a background image region other than the target image region can be determined. A preset blur parameter can be obtained, and the background image region can be processed to blur the image based on the preset blur parameter to obtain the target image.

[0093] Among them, the preset blur parameter refers to the parameter set in advance to blur the image area to achieve the blurring effect. The larger the preset blur parameter, the higher the degree of blurring of the image area can be considered, and the smaller the preset blur parameter, the lower the degree of blurring of the image area can be considered.

[0094] The target image region corresponding to the target can be determined in the captured image using depth image. For example, the target can be a person, and the target image region can be the image region corresponding to the person's image. Of course, the target can also be an animal, a static object, etc., and this example embodiment is not limited to this.

[0095] It can identify the background image area other than the target image area, and perform image blurring processing on the background image area according to preset blur parameters to obtain a target image with a blurred effect, thereby improving the user's shooting experience.

[0096] Optional, can be done through Figure 7 The steps described above achieve image blurring of the background image area. (Refer to...) Figure 7 As shown, it can specifically include:

[0097] Step S710: If it is determined that the occlusion detection result indicates that the second image is occluded, then the transition region between the target image region and the background image region is determined based on the occlusion area.

[0098] Step S720: Adjust the preset blur parameters according to the distance from each pixel in the transition region to the target image region to obtain the target blur parameters;

[0099] Step S730: The transition region is blurred using the target blur parameter, and the background image region other than the transition region is blurred using the preset blur parameter to obtain the target image.

[0100] The transition region refers to the image region at the boundary between the target image region and the background image region. In this embodiment, the transition region is mainly distributed in the background image region. Specifically, the transition region between the target image region and the background image region can be determined by the occlusion area. The larger the occlusion area, the lower the accuracy of the obtained depth image. Correspondingly, the transition region between the target image region and the background image region can be set to be larger to weaken the blurring effect of the target image and reduce the problem of inaccurate blurring effect of the target image due to the depth image. Of course, the smaller the occlusion area, the higher the accuracy of the obtained depth image. Correspondingly, the transition region between the target image region and the background image region can be set to be smaller to enhance the blurring effect of the target image.

[0101] The target blur parameter refers to the blur parameter applied to the transition region, obtained by adjusting a preset blur parameter. Specifically, the target blur parameter can be obtained by adjusting the preset blur parameter according to the distance of each pixel in the transition region to the target image region. For example, the target blur parameter can be obtained by adjusting the preset blur parameter according to the distance of each pixel in the transition region to the geometric center of the target image region, or it can be obtained by adjusting the preset blur parameter according to the distance of each pixel in the transition region to the edge of the target image region. This embodiment is not limited to these methods. The target blur parameter corresponding to pixels in the transition region that are closer to the target image region can be set smaller, and conversely, the target blur parameter corresponding to pixels that are farther away from the target image region can be set larger, until it is consistent with the preset blur parameter of the background image region, thereby achieving a smooth blur effect in the transition region and effectively improving the blur effect quality of the target image.

[0102] In an exemplary embodiment, occlusion warning information can be displayed, which may include occlusion detection results and image blurring result information. The occlusion detection results can be displayed in their entirety, such as showing whether occlusion exists and the occlusion area, or only the occluded content can be displayed. The image blurring result information refers to the strength of the current blurring effect. For example, when the occlusion area is greater than or equal to a target threshold, the accuracy of the obtained depth image may be relatively low; in this case, the blurring effect can be relatively weakened to ensure the accuracy of the image blurring result. Conversely, when the occlusion area is less than the target threshold, the accuracy of the obtained depth image may be relatively high; in this case, the blurring effect can be relatively enhanced to ensure the image blurring effect.

[0103] Specifically, the size of the transition area can be determined based on the occlusion area, and the strength of the blurring effect can be controlled based on the distance of each pixel in the transition area to the target image area. For details, please refer to the aforementioned embodiments, which will not be repeated here.

[0104] By displaying obstruction warnings, the system concisely informs users that the camera is obstructed, while also reminding them that a bokeh effect can still be achieved even with obstruction, although the effect may be weaker, thus effectively improving the user's shooting experience.

[0105] Optionally, if the occlusion detection result indicates that the second image is occluded and the occlusion area is greater than or equal to the target threshold, the displayed image blurring result information can be a level 1 image blurring, which indicates a weak blurring effect; if the occlusion detection result indicates that the second image is occluded and the occlusion area is less than the target threshold, the displayed image blurring result information can be a level 2 image blurring, which indicates a strong blurring effect.

[0106] Figure 8This illustration schematically depicts a display of occlusion warning information in an exemplary embodiment of the present disclosure.

[0107] refer to Figure 8 As shown, if the occlusion detection result indicates that the second image is occluded, and the occlusion area is greater than or equal to the target threshold, occlusion prompt information 801 can be displayed. The occlusion detection result of occlusion prompt information 801 indicates that there is occlusion, and the image blurring result information can be level one image blurring (weak). If the occlusion detection result indicates that the second image is occluded, and the occlusion area is less than the target threshold, occlusion prompt information 802 can be displayed. The occlusion detection result of occlusion prompt information 802 indicates that there is occlusion, and the image blurring result information can be level two image blurring (strong).

[0108] When there is obstruction, displaying image blurring results information can indicate the strength of the current image blurring effect to the user, guiding the user's understanding of the image blurring effect and improving the user experience. To a certain extent, it can also guide the user to achieve different blurring effects by obstructing the camera, increasing the diversity of the target images captured and enhancing the fun of taking photos.

[0109] In summary, this exemplary embodiment can acquire captured images, which are first and second images obtained by different cameras capturing the same target. Occlusion detection can be performed based on the first and second images to determine the occlusion detection result. Then, at least one of the first and second images can be selected for depth estimation based on the occlusion detection result to determine the depth image corresponding to the captured image. Finally, based on the depth image and the captured image, the target image is determined and used as the captured output. By selecting a suitable image as input for depth estimation based on the occlusion detection result, the accuracy of the depth image can be effectively improved even when the camera is occluded, thereby ensuring the image quality of the output target image, such as improving the image blurring effect of the target image.

[0110] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0111] Further reference Figure 9 As shown, this example embodiment also provides an image processing apparatus 900, including an image acquisition module 910, an occlusion detection module 920, a depth estimation module 930, and an image output module 940. Wherein:

[0112] Image acquisition module 910 is used to acquire captured images, including a first image and a second image obtained by different cameras capturing the same target;

[0113] The occlusion detection module 920 is used to perform occlusion detection based on the first image and the second image, and obtain the occlusion detection result;

[0114] The depth estimation module 930 is used to obtain at least one corresponding depth image of the first image and the second image based on the occlusion detection result.

[0115] In an exemplary embodiment, the occlusion detection result may include the presence of occlusion in the second image and the occlusion area when the second image is occluded; the depth estimation module 930 may be used for:

[0116] If the occlusion area of ​​the second image is greater than or equal to the target threshold, then the first image is selected; and

[0117] Based on the first image, monocular depth estimation is performed to obtain the depth image corresponding to the captured image.

[0118] In one exemplary embodiment, the depth estimation module 930 can be used to:

[0119] If the occlusion area of ​​the second image is less than the target threshold, then both the first image and the second image are selected; and

[0120] Based on the first image, perform monocular depth estimation to determine the first depth estimation result;

[0121] Based on the first image and the second image, perform binocular depth estimation to determine the second depth estimation result;

[0122] According to a preset first fusion weight, the first depth estimation result and the second depth estimation result are fused to obtain the depth image corresponding to the captured image.

[0123] In an exemplary embodiment, the occlusion detection result may include the absence of occlusion in the second image; the depth estimation module 930 may be used for:

[0124] If the occlusion detection result indicates that the second image does not have an occlusion, then the first image and the second image are selected;

[0125] Based on the first image and the second image, a binocular depth estimation is performed to determine the depth image corresponding to the captured image.

[0126] In one exemplary embodiment, the occlusion detection module 920 can be used to:

[0127] Determine the statistical histograms corresponding to the first image and the second image respectively;

[0128] Occlusion detection is performed on the first image and the second image based on the statistical histogram to obtain the occlusion detection results.

[0129] In an exemplary embodiment, the statistical histogram may include a grayscale histogram and a gradient histogram; the occlusion detection module 920 may be used for:

[0130] The first correlation coefficient is determined based on the grayscale histograms corresponding to the first image and the second image, respectively.

[0131] The second correlation coefficient is determined based on the gradient histograms corresponding to the first image and the second image, respectively.

[0132] The first correlation coefficient and the second correlation coefficient are fused according to the preset second fusion weight, and the occlusion detection result is determined based on the fusion result.

[0133] In one exemplary embodiment, the occlusion detection module 920 can be used to:

[0134] Determine the first image hash result of the first image, and determine the second image hash result of the second image;

[0135] Calculate the Hamming distance between the hash results of the first image and the hash results of the second image, and determine the image similarity based on the Hamming distance;

[0136] The occlusion detection result is determined by the image similarity.

[0137] In one exemplary embodiment, the image output module 940 can be used to:

[0138] Based on the depth image, a target image region corresponding to the target is determined in the captured image, and a background image region other than the target image region is determined.

[0139] Obtain preset blur parameters, and perform image blurring processing on the background image area according to the preset blur parameters to obtain the target image.

[0140] In one exemplary embodiment, the image output module 940 can be used to:

[0141] If the occlusion detection result indicates that the second image is occluded, then the transition region between the target image region and the background image region is determined based on the occlusion area.

[0142] Based on the distance from each pixel in the transition region to the target image region, the preset blur parameters are adjusted to obtain the target blur parameters;

[0143] The target image is obtained by blurring the transition region using the target blur parameter and by blurring the background image region other than the transition region using the preset blur parameter.

[0144] In one exemplary embodiment, the image processing apparatus 900 further includes an occlusion warning module, which can be used to:

[0145] Display occlusion warning information, which includes occlusion detection results and image blurring results.

[0146] In one exemplary embodiment, the occlusion warning module can be used to:

[0147] If the occlusion detection result indicates that the second image is occluded and the occlusion area is greater than or equal to the target threshold, then the displayed image blurring result information is a level of image blurring.

[0148] If the occlusion detection result indicates that the second image is occluded and the occlusion area is less than the target threshold, then the displayed image blurring result information is level two image blurring.

[0149] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0150] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0151] Exemplary embodiments of this disclosure also provide an electronic device. This electronic device may be a terminal device among the aforementioned terminal devices 110, 120, and camera system 130. Generally, the electronic device may include a processor and a memory, the memory being used to store executable instructions of the processor, and the processor being configured to perform the aforementioned image processing method by executing the executable instructions.

[0152] The following is based on Figure 10 Taking a mobile terminal 1000 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 10 The structure can also be applied to fixed types of equipment.

[0153] like Figure 10 As shown, the mobile terminal 1000 may specifically include: a processor 1001, a memory 1002, a bus 1003, a mobile communication module 1004, an antenna 1, a wireless communication module 1005, an antenna 2, a display screen 1006, a camera module 1007, an audio module 1008, a power module 1009, and a sensor module 1010.

[0154] Processor 1001 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The image processing method in this exemplary embodiment can be executed by an AP, GPU, or DSP. When the method involves neural network-related processing, it can be executed by an NPU. For example, the NPU can load neural network parameters and execute neural network-related algorithm instructions.

[0155] An encoder encodes (compresses) images or videos to reduce data size for easier storage or transmission. A decoder decodes (decompresses) the encoded data to restore the original image or video data. The mobile terminal 1000 can support one or more encoders and decoders, such as image formats like JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), and BMP (Bitmap), and video formats like MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).

[0156] The processor 1001 can be connected to the memory 1002 or other components via the bus 1003.

[0157] The memory 1002 can be used to store computer executable program code, which includes instructions. The processor 1001 executes various functional applications and data processing of the mobile terminal 1000 by running the instructions stored in the memory 1002. The memory 1002 can also store application data, such as images, videos, and other files.

[0158] The communication function of mobile terminal 1000 can be implemented through mobile communication module 1004, antenna 1, wireless communication module 1005, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1004 can provide 3G, 4G, 5G and other mobile communication solutions for mobile terminal 1000. Wireless communication module 1005 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1000.

[0159] The display screen 1006 is used to implement display functions, such as displaying the user interface, images, and videos. The camera module 1007 is used to implement shooting functions, such as capturing images and videos. The audio module 1008 is used to implement audio functions, such as playing audio and capturing voice. The power module 1009 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring battery status.

[0160] The sensor module 1010 may include one or more sensors to implement corresponding sensing and detection functions. For example, the sensor module 1010 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1000 and output inertial sensing data.

[0161] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0162] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0163] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0164] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0165] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0166] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Acquire captured images, including a first image and a second image obtained by capturing the same target with different cameras; Occlusion detection is performed based on the first image and the second image to obtain occlusion detection results; Based on the occlusion detection results, at least one corresponding depth image of the first image and the second image is obtained, including: when the occlusion area of ​​the second image is greater than or equal to the target threshold, the first image is selected; as well as Based on the first image, monocular depth estimation is performed to obtain the depth image corresponding to the captured image.

2. The method according to claim 1, characterized in that, The step of obtaining at least one corresponding depth image from the first image and the second image based on the occlusion detection result includes: If the occlusion area of ​​the second image is less than the target threshold, then both the first image and the second image are selected; and Based on the first image, perform monocular depth estimation to determine the first depth estimation result; Based on the first image and the second image, perform binocular depth estimation to determine the second depth estimation result; According to a preset first fusion weight, the first depth estimation result and the second depth estimation result are fused to obtain the depth image corresponding to the captured image.

3. The method according to claim 1, characterized in that, The occlusion detection result includes that the second image is not occluded; the step of obtaining at least one corresponding depth image of the first image and the second image based on the occlusion detection result includes: If the occlusion detection result indicates that the second image does not have an occlusion, then the first image and the second image are selected; Based on the first image and the second image, a binocular depth estimation is performed to determine the depth image corresponding to the captured image.

4. The method according to claim 1, characterized in that, The step of performing occlusion detection based on the first image and the second image to obtain occlusion detection results includes: Determine the statistical histograms corresponding to the first image and the second image respectively; Occlusion detection is performed on the first image and the second image based on the statistical histogram to obtain the occlusion detection results.

5. The method according to claim 4, characterized in that, The statistical histograms include gray-level histograms and gradient histograms; The step of performing occlusion detection on the first image and the second image based on the statistical histogram and obtaining the occlusion detection result includes: The first correlation coefficient is determined based on the grayscale histograms corresponding to the first image and the second image, respectively. The second correlation coefficient is determined based on the gradient histograms corresponding to the first image and the second image, respectively. The first correlation coefficient and the second correlation coefficient are fused according to the preset second fusion weight, and the occlusion detection result is determined based on the fusion result.

6. The method according to claim 1, characterized in that, The step of performing occlusion detection based on the first image and the second image to obtain occlusion detection results includes: Determine the first image hash result of the first image, and determine the second image hash result of the second image; Calculate the Hamming distance between the hash results of the first image and the hash results of the second image, and determine the image similarity based on the Hamming distance; The occlusion detection result is determined by the image similarity.

7. The method according to claim 1, characterized in that, The method further includes: Based on the depth image, a target image region corresponding to the target being captured is determined in the captured image, and a background image region other than the target image region is determined. Obtain preset blur parameters, and perform image blurring processing on the background image area according to the preset blur parameters to obtain the target image.

8. The method according to claim 7, characterized in that, The step of performing image blurring processing on the background image region according to the blurring parameters to obtain the target image includes: If the occlusion detection result indicates that the second image is occluded, then the transition region between the target image region and the background image region is determined based on the occlusion area. Based on the distance from each pixel in the transition region to the target image region, the preset blur parameters are adjusted to obtain the target blur parameters; The target image is obtained by blurring the transition region using the target blur parameter and by blurring the background image region other than the transition region using the preset blur parameter.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Display occlusion warning information, which includes occlusion detection results and image blurring results.

10. The method according to claim 9, characterized in that, The method further includes: If the occlusion detection result indicates that the second image is occluded and the occlusion area is greater than or equal to the target threshold, then the displayed image blurring result information is a level of image blurring. If the occlusion detection result indicates that the second image is occluded and the occlusion area is less than the target threshold, then the displayed image blurring result information is level two image blurring.

11. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire captured images, including a first image and a second image obtained by capturing the same target with different cameras; An occlusion detection module is used to perform occlusion detection based on the first image and the second image, and obtain occlusion detection results; The depth estimation module is used to obtain at least one corresponding depth image of the first image and the second image based on the occlusion detection result, including: selecting the first image when the occlusion area of ​​the second image is greater than or equal to a target threshold; Additionally, monocular depth estimation is performed based on the first image to obtain a depth image corresponding to the captured image.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 10.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 10 by executing the executable instructions.

Images