Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium. The method comprises the steps that a first scene image is acquired; obtaining monocular depth information and scene depth information related to the first scene image; and processing the first scene image according to the monocular depth information and the scene depth information to obtain a target image. According to the invention, the depth calculation accuracy can be effectively improved, and the blurring rendering effect of the shot scene image is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of electronic devices, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] With the development of smart terminals, the requirements for the camera functions of smart terminals are getting higher and higher. Due to the portability and cost requirements of smart terminals, they often carry smaller imaging sensors, which makes it difficult for smart terminals to capture images with blur effects. In related technologies, when a target object (such as a person) in a shooting scene is photographed at a close-up level, a depth constraint is imposed on the target object to maintain a clear image area of ​​the target object.

[0003] In this way, the depth calculation accuracy is not high, and the blurred rendering effect of the captured scene image is not good. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the present disclosure proposes an image processing method, device, electronic device, non-transitory computer-readable storage medium storing computer instructions, and computer program product, which can effectively improve the accuracy of depth calculation and enhance the blurred rendering effect of the captured scene image.

[0006] The image processing method proposed in the first aspect of the embodiment of the present disclosure includes: acquiring a first scene image; acquiring monocular depth information and scene depth information related to the first scene image; and processing the first scene image according to the monocular depth information and the scene depth information to obtain a target image.

[0007] The image processing device proposed in the second aspect of the embodiment of the present disclosure includes: a first acquisition module, used to acquire a first scene image; a second acquisition module, used to acquire monocular depth information and scene depth information related to the first scene image; and a processing module, used to process the first scene image according to the monocular depth information and the scene depth information to obtain a target image.

[0008] The electronic device proposed in the embodiment of the third aspect of the present disclosure includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method proposed in the above embodiment of the present disclosure is implemented.

[0009] The fourth aspect of the present disclosure provides a non-temporary computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in the above embodiment of the present disclosure.

[0010] The fifth aspect of the present disclosure provides a computer program product. When the instructions in the computer program product are executed by a processor, the method provided in the above embodiment of the present disclosure is performed.

[0011] The image processing method, device, electronic device, non-transitory computer-readable storage medium storing computer instructions, and computer program product provided by the present disclosure can effectively improve the accuracy of depth calculation and improve the blur rendering effect of the captured scene image by acquiring a first scene image, monocular depth information and scene depth information related to the first scene image, and processing the first scene image according to the monocular depth information and the scene depth information to obtain a target image.

[0012] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0014] Figure 1 is a flowchart of an image processing method proposed in an embodiment of the present disclosure;

[0015] Figure 2 is a flowchart of an image processing method proposed in another embodiment of the present disclosure;

[0016] Figure 3 is a flowchart of an image processing method proposed in another embodiment of the present disclosure;

[0017] Figure 4 is a flowchart of an image processing method proposed in another embodiment of the present disclosure;

[0018] Figure 5 is a flowchart of an image processing method proposed in yet another embodiment of the present disclosure;

[0019] Figure 6 is a structural schematic diagram of an image processing device proposed in an embodiment of the present disclosure;

[0020] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present disclosure, and are not to be construed as limitations of the present disclosure. On the contrary, the embodiments of the present disclosure include all changes, modifications, and equivalents that fall within the spirit and connotation of the appended claims.

[0022] Figure 1 It is a flowchart of an image processing method proposed in an embodiment of the present disclosure.

[0023] In this embodiment, the image processing method is configured as an image processing device. In this embodiment, the image processing method can be configured in the image processing device, and the image processing device can be set in a terminal. The terminal includes, for example, a mobile phone, a wearable device, an Internet of Things device, a car with a communication function, a smart car, a tablet computer, a computer with a wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in a smart grid, a wireless terminal device in transportation safety, a wireless terminal device in a smart city, and a wireless terminal device in a smart home, but is not limited thereto.

[0024] It should be noted that the execution subject of the embodiment of the present disclosure may be, for example, a central processing unit (CPU) of a terminal or server in terms of hardware, and may be, for example, a related background service of a terminal or server in terms of software, without limitation. The terminal in the embodiment of the present disclosure may be equipped with multiple camera devices, each of which may be used to capture a scene to obtain a scene image.

[0025] like Figure 1 As shown, the image processing method comprises:

[0026] S101: Acquire a first scene image.

[0027] In some embodiments, the first scene image may be, for example, a frame of scene image captured by a camera device in the terminal. For example, a main camera may be used to capture a scene to obtain the first scene image.

[0028] In the disclosed embodiment, a local area image in the first scene image may be subjected to blur rendering processing to simulate an extremely shallow depth of field rendering effect of a large aperture lens, thereby improving the blur rendering effect.

[0029] S102: Obtain monocular depth information and scene depth information related to the first scene image.

[0030] Among them, monocular depth information refers to the depth information obtained by a single camera or image sensor with only one perspective. The scene depth image contains the depth information of the shooting scene obtained based on multiple cameras or image sensors. For example, when there are two cameras, two frames of scene images are captured based on the two cameras. The depth information that can be captured by the two cameras can also be called binocular depth information. Binocular depth information can be an optional example of scene depth information.

[0031] In some embodiments, the monocular depth information and scene depth information related to the first scene image can be obtained based on artificial intelligence (AI); or the monocular depth information and scene depth information related to the first scene image can be obtained based on a depth information analysis algorithm; or any other possible method can be used to obtain the monocular depth information and scene depth information related to the first scene image, without limitation.

[0032] S103: Processing the first scene image according to the monocular depth information and the scene depth information to obtain a target image.

[0033] In some embodiments, the monocular depth information and the scene depth information may be fused, and the depth information obtained by the fusion process may be used as the target depth information, and the target depth information may be used to render the first scene image to obtain the target image; alternatively, the monocular depth information and the scene depth information may be input into a depth information processing model, and the target depth information output by the depth information processing model may be obtained, and the first scene image may be rendered using the target depth information to obtain the target image; of course, any other possible method may be used to process the first scene image according to the monocular depth information and the scene depth information to obtain the target image, and there is no limitation to this.

[0034] After obtaining the monocular depth information and the scene depth information, the monocular depth information and the scene depth information can be used to render the first scene image. When the monocular depth information and the scene depth information are used to jointly characterize the depth distribution of the shooting scene, good depth calculation accuracy can be achieved. When the first scene image is rendered based on the monocular depth information and the scene depth information, the focused part in the first scene image can be regarded as the focal plane, and other depths outside the focal plane in the first scene image can be blurred to obtain a target image with a better blur effect. As a result, the depth calculation accuracy can be effectively improved, and the blurred rendering effect of the captured scene image can be improved.

[0035] In this embodiment, by acquiring a first scene image, monocular depth information and scene depth information related to the first scene image, and processing the first scene image according to the monocular depth information and the scene depth information, a target image is obtained, which can effectively improve the accuracy of depth calculation and improve the blurred rendering effect of the captured scene image.

[0036] Figure 2 It is a flowchart of an image processing method proposed in another embodiment of the present disclosure.

[0037] like Figure 2 As shown, the image processing method comprises:

[0038] S201: Capturing multiple frames of scene images based on multiple camera devices, respectively, wherein a first scene image belongs to the multiple frames of scene images.

[0039] In some embodiments, each camera device may be used to shoot a scene to obtain a frame of scene image. Accordingly, multiple frames of scene images may be captured based on multiple camera devices. The scene image captured by the main camera device in the multiple frames of scene images may be used as the first scene image.

[0040] In the embodiment of the present disclosure, a dual camera device may be installed in the terminal, and a clear dual camera image may be captured based on the dual camera device. The dual camera device may include a main camera device and a secondary camera device, and the dual camera image may also be a clear main camera image and a secondary camera image after calibration, the main camera image may be captured by the main camera device, and the secondary camera image may be captured by the secondary camera device. Then, corresponding depth processing is performed based on the main camera image and the secondary camera image.

[0041] S202: Determine monocular depth information corresponding to each frame of the scene image, and use the multiple monocular depth information as the monocular depth information related to the first scene image.

[0042] S203: Determine scene depth information related to the first scene image according to the multiple frames of scene images.

[0043] In some embodiments, depth recognition may be performed on each scene image to obtain corresponding monocular depth information; or each scene image may be processed using a feature-based method and a deep learning-based method to obtain monocular depth information; or monocular depth information may be obtained based on any other possible method.

[0044] In some embodiments, when performing the step of determining the monocular depth information corresponding to each scene image, the monocular depth information corresponding to each scene image may be determined based on an AI monocular depth algorithm, thereby effectively improving the accuracy and efficiency of determining the monocular depth information.

[0045] For example, considering that the texture information inside the portrait is complex, with many weak and / or repeated textures. In order to better simulate the shallow depth of field effect of the camera's large aperture lens, the monocular depth information can be generated based on the AI ​​monocular depth algorithm, and the monocular depth information and binocular depth information (an optional example of scene depth information) are fused and calculated in the subsequent algorithm flow, thereby improving the accuracy of the depth calculation inside the portrait.

[0046] Among them, scene depth information may also refer to multi-eye depth information. Multi-eye depth information refers to images with depth information from different perspectives obtained through multiple cameras or image sensors. Scene depth information includes depth information of the shooting scene obtained based on multiple cameras or image sensors. Taking the example of two frames of scene images, which are respectively acquired based on two corresponding camera devices, the scene depth information may be, for example, binocular depth information. Binocular depth information refers to images with depth information from different perspectives obtained through two cameras or image sensors.

[0047] In some embodiments, multiple frames of scene images may be processed based on a parallax method to obtain scene depth information; or multiple frames of scene images may be processed based on a structured light method to obtain scene depth information; or multiple frames of scene images may be processed based on a time of flight (TOF) method to obtain scene depth information, without limitation.

[0048] In some embodiments, if the number of the multiple camera devices is two, two frames of scene images may be processed based on a dual-camera stereo matching algorithm to determine scene depth information.

[0049] S204: Determine a regional image corresponding to the scene image, wherein different regional images contain the same target object.

[0050] After collecting multiple frames of scene images, local area images can be identified from each frame of scene images, and different area images contain the same target object. The target object is, for example, a person. In the embodiment of the present disclosure, a person in a scene is photographed, and the blurred rendering effect of the image containing the person area obtained by the photographing is processed as an example.

[0051] It is understandable that the scene image may include a regional image of a person and a regional image of a background. In the disclosed embodiment, a person in each frame of the scene image may be identified, and the regional image where the person is located may be used as the regional image corresponding to the scene image.

[0052] In some embodiments, each frame of the scene image can be processed based on an image processing algorithm to obtain a regional image where the person in the scene image is located; or the person's features can be first identified from the scene image, and the regional image can be determined from the scene image based on the features of the identified person; or the regional image corresponding to the scene image can be determined based on any other possible method, such as a machine learning method, a mathematical algorithm method, etc.

[0053] In the disclosed embodiment, a region image containing a target object can be detected from a scene image, thereby effectively improving the accuracy and efficiency of region image detection and recognition.

[0054] In some embodiments, multiple pixels in the scene image can be detected pixel by pixel to determine whether each pixel point is divided into a local area image of the person, and then the local area image of the person is determined based on the determination result corresponding to each pixel point.

[0055] In some embodiments, when executing the step of detecting a regional image containing a target object from a scene image, candidate pixel features of each candidate pixel in the scene image can be determined, and reference pixel features related to the target object can be determined, wherein the reference pixel features include: at least one object feature of the target object is mapped to a pixel feature in a pixel dimension, the candidate pixel features and the reference pixel features are processed to determine a probability value that the candidate pixel belongs to the regional image, and based on multiple probability values, the regional image containing the target object is detected from the scene image, thereby further improving the accuracy and efficiency of regional image detection and recognition.

[0056] Among them, the scene image will contain multiple pixels, each pixel can be called a candidate pixel, and each candidate pixel will correspond to some features, which can be called candidate pixel features. Candidate pixel features can be, for example, color values, gray values, depth values, semantic features, etc. of candidate pixels, and there is no limitation on this. In some embodiments, reference pixel features related to the target object can also be determined. Reference pixel features can be understood as pixel features of reference pixels in the image obtained by photographing the target object. The reference pixel features can include multiple pixel features, each of which is an object feature of the target object mapped to the pixel dimension. Object features are, for example, features of the target object in real space, such as skin color, facial features, hair color, etc. Mapped to the pixel dimension, it can be understood as pixel features corresponding to skin color, pixel features corresponding to facial features, pixel features corresponding to hair color, etc. in the image obtained by photographing the target object in real space.

[0057] In some embodiments, after identifying the candidate pixel features of each candidate pixel in the scene image, the candidate pixel features and the reference pixel features can be processed based on AI to determine the probability value that the candidate pixel belongs to the regional image. For example, the candidate pixel features and the reference pixel features can be processed based on an AI model to obtain corresponding probability values, and the AI ​​model has learned the mapping relationship between the candidate pixel features and the reference pixel features, and the probability values ​​that the corresponding candidate pixel belongs to the regional image. Then, based on multiple probability values, the regional image containing the target object can be detected from the scene image.

[0058] In some embodiments, when performing the step of detecting a region image containing a target object from a scene image according to multiple probability values, partial probability values ​​greater than or equal to a probability threshold may be determined from the multiple probability values, and a region image may be generated according to candidate pixel points corresponding to each probability value in the partial probability values, thereby improving the generation effect and accuracy of the region image, as well as the convenience of generation.

[0059] The probability threshold may be a threshold value for determining the probability value of a candidate pixel belonging to the regional image. When the probability value is greater than or equal to the probability threshold, it indicates that the corresponding candidate pixel belongs to a pixel in the regional image and can be classified into the regional image. When the probability value is less than the probability threshold, it indicates that the corresponding candidate pixel does not belong to a pixel in the regional image and may not be classified into the regional image.

[0060] After determining the partial probability values ​​greater than or equal to the probability threshold from the multiple probability values, the regional image can be synthesized according to the candidate pixel points corresponding to each probability value in the partial probability values.

[0061] In the disclosed embodiment, each frame of scene image can be identified to obtain a regional image corresponding to each frame of scene image. Then, multiple regional images can be obtained based on multiple frames of scene images, which can improve the accuracy of region detection. When the target image is realized based on multiple regional images, the generated target image can have more layering and better image expression effect.

[0062] For example, taking the camera device as a mobile phone camera, the portrait mode of the mobile phone camera can be turned on to shoot the subject portrait, in order to accurately simulate the imaging effect of the shallow depth of field optical lens, so as to accurately calculate the depth information of the subject portrait. The portrait area can be specially processed, and the accurate portrait area (regional image of the portrait) in the scene image can be first extracted. The portrait area can be accurately detected pixel by pixel from the input image (multi-frame scene image). For example, the portrait area can be detected pixel by pixel through the AI ​​algorithm.

[0063] S205: Processing the first scene image according to the multiple region images, the monocular depth information, and the scene depth information to obtain a target image.

[0064] After determining the multiple regional images, monocular depth information and scene depth information, the first scene image can be rendered according to the multiple regional images, monocular depth information and scene depth information, and the image obtained by the rendering process is used as the target image.

[0065] In this embodiment, multiple frames of scene images are collected based on multiple camera devices, and the monocular depth information corresponding to each frame of the scene image is determined, and multiple monocular depth information is used as the monocular depth information related to the first scene image, and the scene depth information related to the first scene image is determined based on the multiple frames of scene images, and the regional image corresponding to the scene image is determined, wherein different regional images contain the same target object, and the first scene image is processed based on the multiple regional images, the monocular depth information, and the scene depth information to obtain the target image. The accuracy of depth calculation can be greatly improved, and the blurred rendering effect of the local regional image of the target object in the captured scene image can be improved.

[0066] Figure 3 It is a flowchart of an image processing method proposed in another embodiment of the present disclosure.

[0067] like Figure 3 As shown, the image processing method comprises:

[0068] S301: Capturing multiple frames of scene images based on multiple camera devices, respectively, wherein a first scene image belongs to the multiple frames of scene images.

[0069] For the description of S301 , please refer to the above embodiment, which will not be repeated here.

[0070] Taking a portrait as the target object as an example, if multiple portraits are detected in a scene image, a selection interface can be provided to allow the user to select one or part of the portraits to be blurred and rendered. It can also automatically perform blurred rendering on only one of the portraits, such as selecting the C-position portrait for blurred rendering, or selecting the front portrait for blurred rendering. It can also support blurred rendering of multiple portraits without any restriction.

[0071] S302: Inputting a scene image into a depth estimation model, wherein the depth estimation model is used to identify feature information from the scene image and predict a first scene depth of each pixel in the scene image based on the feature information.

[0072] In some embodiments, feature information, such as the grayscale value of the scene image or any other possible feature. The depth corresponding to each pixel in the scene image can be referred to as the first scene depth. Depth refers to the distance between a point in an object or scene and the surface of the object or the scene reference plane. In computer vision and graphics, depth is usually expressed as a depth value for each pixel or a depth image.

[0073] Among them, the depth estimation model can be, for example, an AI model, such as a neural network model or a machine learning model. The depth estimation model can be pre-trained. For example, the depth estimation model can be a depth value corresponding to each pixel in a single scene image that is input (an optional example of the first scene depth). The depth estimation model can obtain the corresponding depth value based on the grayscale value or feature of each pixel in the image. In the training stage, a large amount of annotated data can be used, including images with depth information and corresponding depth images. Through the training of the model, the trained depth estimation model learns the relationship between the visual features and the depth in the image, so that the depth of the new image can be estimated.

[0074] S303: Acquire multiple first scene depths output by the depth estimation model, and use the multiple first scene depths as monocular depth information.

[0075] After obtaining the multiple first scene depths output by the depth estimation model, the multiple first scene depths can be used as monocular depth information.

[0076] S304: Determine a scene disparity according to the multiple frames of scene images, wherein the scene disparity represents a horizontal offset of the captured scene on pixel planes of different camera devices.

[0077] In an embodiment of the present disclosure, scene depth information can also be formed based on multiple frames of scene images. It should be noted that the second scene depth of the captured scene contained in the scene depth information is not the same as the first scene depth contained in the monocular depth information. The first scene depth is the scene depth determined based on a single camera device, while the second scene depth refers to the scene depth determined based on the parallax between multiple camera devices. Therefore, in the embodiment of the present disclosure, the scene depths of multiple focal planes can be learned, and then the blurred rendering effect of the captured scene image can be improved based on the scene depths of multiple focal planes.

[0078] In some embodiments, each frame of scene image may be subjected to parallax analysis processing to determine the horizontal offset of the captured scene on the pixel planes of different camera devices. For example, assuming that there are three frames of scene images, scene image A may be analyzed to determine the parallax A of the captured scene on the pixel plane of camera device A that collects scene image A, and scene image B may be analyzed to determine the parallax B of the captured scene on the pixel plane of camera device A that collects scene image B, and scene image C may be analyzed to determine the parallax C of the captured scene on the pixel plane of camera device C that collects scene image C, and then at least two of the parallax A, parallax B, and parallax C may be subjected to parallax comparison to determine the horizontal offset of the captured scene on the pixel planes of different camera devices. The number of scene parallaxes may be one or more, and different scene parallaxes represent the horizontal offset of the captured scene on the pixel planes of different camera devices under the corresponding camera device combination under different camera device combinations.

[0079] In some embodiments, at least two of the above-mentioned disparity A, disparity B, and disparity C may be disparities obtained by analyzing scene images collected at the same time. For example, if the second scene depth needs to be analyzed based on disparity A and disparity B, the scene image A and scene image B corresponding to disparity A and disparity B are collected at the same time. And so on.

[0080] S305: Determine a second scene depth of the captured scene according to the scene parallax, and use the second scene depth as scene depth information.

[0081] After the horizontal offsets of the shooting scenes on the pixel planes of different camera devices are determined and used as the scene parallax, the scene parallax can be solved to obtain the second scene depth.

[0082] After the plurality of second scene depths output by the depth estimation model are obtained, the second scene depths may be used as scene depth information.

[0083] S306: Determine a regional image corresponding to the scene image, wherein different regional images contain the same target object.

[0084] S307: Process the first scene image according to the multiple region images, the monocular depth information, and the scene depth information to obtain a target image.

[0085] After determining the multiple regional images, monocular depth information and scene depth information, the first scene image can be rendered according to the multiple regional images, monocular depth information and scene depth information, and the image obtained by the rendering process is used as the target image.

[0086] In this embodiment, multiple frames of scene images are collected based on multiple camera devices, and the monocular depth information corresponding to each scene image is determined. According to the multiple frames of scene images, the scene depth image is determined, and the regional image corresponding to the scene image is determined, wherein different regional images contain the same target object, and the first scene image is processed according to the multiple regional images, the monocular depth information and the scene depth information to obtain the target image. The accuracy of depth calculation can be effectively improved, and the blur rendering effect of the captured scene image can be improved. By inputting the scene image into the depth estimation model, wherein the depth estimation model is used to identify feature information from the scene image, and predict the first scene depth of each pixel in the scene image according to the feature information, and obtain multiple first scene depths output by the depth estimation model, and use the multiple first scene depths as monocular depth information. Thereby, the accuracy and efficiency of determining the monocular depth information are effectively improved. By determining the scene parallax according to the multiple frames of scene images, wherein the scene parallax represents the horizontal offset of the captured scene on the pixel planes of different camera devices, and determining the second scene depth of the captured scene according to the scene parallax, and using the second scene depth as the scene depth information. Thereby, the accuracy and efficiency of determining the scene depth information are effectively improved.

[0087] Figure 4 It is a flowchart of an image processing method proposed in another embodiment of the present disclosure.

[0088] like Figure 4 As shown, the image processing method comprises:

[0089] S401: Capturing multiple frames of scene images based on multiple camera devices, respectively, wherein a first scene image belongs to the multiple frames of scene images.

[0090] S402: Determine monocular depth information corresponding to each frame of the scene image, and use the multiple monocular depth information as the monocular depth information related to the first scene image.

[0091] S403: Determine scene depth information related to the first scene image according to the multiple frames of scene images.

[0092] S404: Determine a regional image corresponding to the scene image, wherein different regional images contain the same target object.

[0093] For the description of S401 - S404 , please refer to the above embodiment for details, which will not be repeated here.

[0094] S405: Generate a monocular depth image according to the monocular depth information, and generate a scene depth image according to the scene depth information.

[0095] In some embodiments, a depth image formed based on monocular depth information may be referred to as a monocular depth image. A depth image formed based on scene depth information may be referred to as a scene depth image. A depth image may visually display the corresponding depth information.

[0096] S406: Determine, according to the multiple region images, a monocular effective region image in the monocular depth image and a scene effective region image in the scene depth image.

[0097] The monocular effective area image refers to the local image corresponding to the area image in the monocular depth image. The monocular effective area image can also be understood as the local area where the algorithm takes effect.

[0098] In some embodiments, relative position information of at least one area image in a corresponding scene image may be determined, and a local image indicated by the relative position information may be determined from a monocular depth image, and the local image may be used as a monocular effective area image.

[0099] The scene effective area image refers to a local image in the scene depth image that corresponds to the area image. The scene effective area image can also be understood as the local area where the algorithm takes effect.

[0100] In some embodiments, relative position information of at least one area image in a corresponding scene image may be determined, and a local image indicated by the relative position information may be determined from the scene depth image, and the local image may be used as a scene effective area image.

[0101] S407: Obtaining monocular feature information of the monocular effective area image and scene feature information of the scene effective area.

[0102] In some embodiments, the color information and / or spatial domain information of the monocular effective area image may be obtained, and the color information and / or spatial domain information of the monocular effective area image may be used as the monocular feature information. The color information and / or spatial domain information of the scene effective area may be obtained, and the color information and / or spatial domain information of the scene effective area may be used as the scene feature information.

[0103] In some embodiments, the color information may be, for example, RGB. RGB is a commonly used model for representing color, which represents a combination of three basic colors of red, green, and blue. The RGB model is based on the principle of additive color mixing, and various colors can be obtained by adjusting the combination of red, green, and blue light of different intensities. In the RGB model, the color of each pixel is represented by three components: red (R), green (G), and blue (B). The value range of each component is generally 0 to 255, where 0 represents the absence of light of that color and 255 represents the light of maximum intensity. Various colors can be obtained by adjusting the values ​​of the three components of RGB. For example, pure red can be represented as (255, 0, 0), pure green can be represented as (0, 255, 0), and pure blue can be represented as (0, 0, 255). Other colors are composed of combinations of red, green, and blue light of different intensities.

[0104] In some embodiments, spatial domain information, in image or signal processing, refers to information stored in the spatial domain. The spatial domain refers to the value or feature at each pixel or sampling point, which describes the changes of the image or signal in the time domain and spatial domain. In image processing, spatial domain information includes information such as the pixel value, color, grayscale, etc. of the image, as well as the spatial relationship between them, such as the relative position of pixels, local texture features between neighboring pixels, etc.

[0105] S408: According to the monocular feature information and the scene feature information, the monocular depth image and the scene depth image are fused to obtain a target depth image.

[0106] In some embodiments, the color information of the monocular depth image and the color information of the scene depth image can be fused to obtain a target depth image; or, the spatial domain information of the monocular depth image and the spatial domain information of the scene depth image can be fused to obtain a target depth image; or, the color information and spatial domain information of the monocular depth image, and the color information and spatial domain information of the scene depth image can be fused to obtain a target depth image. As a result, the target depth image can carry more accurate depth information, thereby greatly improving the accuracy of depth calculation. It can also ensure that a smooth and accurate depth transition is generated in the regional image of the portrait inside the target image to support the image rendering requirements of the extremely shallow depth of field algorithm.

[0107] S409: synthesizing the target image according to the target depth image and the first scene image.

[0108] In some embodiments, the depth of the first scene image can be rendered with reference to the target depth image, and the rendered image can be used as the target image; the target depth image and the first scene image can also be input into an image synthesis model, and the output image of the image synthesis model can be used as the target image, and the image synthesis model has learned the mapping relationship between the target depth image, the first scene image and the target image; or the first scene image can be processed based on the target depth image in combination with any other possible method to obtain the target image.

[0109] In this embodiment, multiple frames of scene images are collected based on multiple camera devices, and the monocular depth information corresponding to each scene image is determined. According to the multiple frames of scene images, the scene depth image is determined, and the regional image corresponding to the scene image is determined, wherein different regional images contain the same target object, and the first scene image is processed according to the multiple regional images, the monocular depth information and the scene depth information to obtain the target image. It can effectively improve the accuracy of depth calculation and improve the blur rendering effect of the scene image obtained by shooting. By determining the monocular effective regional image in the monocular depth image according to the multiple regional images, and determining the scene effective regional image in the scene depth image according to the multiple regional images, the monocular feature information of the monocular effective regional image is obtained, and the scene feature information of the scene effective area is obtained, and the monocular depth image and the scene depth image are fused according to the monocular feature information and the scene feature information to obtain the target depth image, and the target image is synthesized according to the target depth image and the first scene image. The target depth image can carry more accurate depth information, thereby greatly improving the accuracy of depth calculation. It can also ensure smooth and accurate depth transition in the image of the portrait area inside the target image to support the image rendering requirements of the extremely shallow depth of field algorithm.

[0110] Figure 5 FIG. 1 is a flow chart of an image processing method proposed in another embodiment of the present disclosure. Figure 5 As shown, a multi-frame scene image includes: main and secondary camera images. In the embodiment of the present disclosure, in order to achieve the effect of shooting close-up portraits with a large aperture and shallow depth of field in the optical lens in mobile phone photography, the relative depth between the focal planes in the picture can be calculated, and the objects in the picture can be rendered according to the different depths of the objects, thereby simulating the shallow depth of field effect in the optical lens. An example is given below:

[0111] Stage 1: Input the clear image to be processed (an optional example of a multi-frame scene image). Since the depth of field of the mobile phone camera is relatively shallow, the input image must be a clear image to ensure the effect of subsequent image processing. In order to simulate the depth of field effect of the optical lens, the depth information of the current scene can be calculated. The input image can be a clear primary and secondary camera image after calibration.

[0112] Stage 2: Since the most common subject in the portrait mode of mobile phone cameras is the portrait, in order to accurately simulate the imaging effect of the shallow depth of field optical lens, the depth information of the scene objects can be accurately calculated. Therefore, the portrait area can be specially processed to extract the accurate portrait area in the main and secondary camera images. For example, the portrait area can be accurately detected pixel by pixel from the input image. For example, the portrait area is detected pixel by pixel by the AI ​​algorithm.

[0113] Stage 3: The number of people in the image can be determined, and the image area corresponding to one of the people is blurred.

[0114] Stage 4: Due to the complex texture information inside the portrait, there are many weak and / or repeated textures. In order to better simulate the shallow depth of field effect of the camera's large aperture lens, a monocular depth image can be generated based on the AI ​​monocular depth algorithm, and the monocular and binocular depth images are fused and calculated in the subsequent algorithm process, thereby improving the accuracy of the depth calculation inside the portrait.

[0115] Stage 5: The depth information in the scene can be calculated based on the dual-camera stereo matching algorithm (an optional example of scene depth information). This step is the dual-camera depth calculation module.

[0116] Stage6: Based on the scene dual-camera depth map output by stage5 (an optional example of a scene depth image), the monocular depth image obtained by the AI ​​algorithm in stage4, and the regional image of the portrait obtained by the AI ​​algorithm in stage2. In this Stage6, the effective area of ​​the algorithm can be first determined by the portrait region map; then, the monocular depth image and the binocular depth image are fused by fusing the color information and spatial domain information in the effective area of ​​the image; finally, the depth image of the entire image is output (an optional example of a target depth image), and the effective area of ​​the algorithm in the portrait area of ​​the depth image is guaranteed to have a smooth and accurate depth transition to support the image rendering requirements of the extremely shallow depth of field algorithm.

[0117] Stage 7: Based on the depth image output by stage 6, the portrait in the main camera image is rendered according to its corresponding depth value. In order to better simulate the optical blur effect, the local area of ​​the portrait obtained by stage 2 after accurate segmentation can be used as a reference. This ensures that the focus plane area of ​​the main portrait in the scene is clear, while other focus planes have accurate rendering effects.

[0118] Figure 6 It is a structural schematic diagram of an image processing device proposed in an embodiment of the present disclosure.

[0119] like Figure 6 As shown, the image processing device 60 includes:

[0120] The first acquisition module 601 is used to acquire a first scene image.

[0121] The second acquisition module 602 is used to acquire monocular depth information and scene depth information related to the first scene image.

[0122] The processing module 603 is used to process the first scene image according to the monocular depth information and the scene depth information to obtain a target image.

[0123] It should be noted that the above explanation of the image processing method is also applicable to the image processing device of this embodiment and will not be repeated here.

[0124] In this embodiment, by acquiring a first scene image, monocular depth information and scene depth information related to the first scene image, and processing the first scene image according to the monocular depth information and the scene depth information, a target image is obtained, which can effectively improve the accuracy of depth calculation and improve the blurred rendering effect of the captured scene image.

[0125] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 7 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0126] like Figure 7 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a memory 28, and a bus 18 that connects various system components (including the memory 28 and the processing unit 16).

[0127] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0128] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0129] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive").

[0130] although Figure 7 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (hereinafter referred to as: CD-ROM), a digital versatile disc read only memory (hereinafter referred to as: DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0131] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0132] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable the human body to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. In addition, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0133] The processing unit 16 executes various functional applications and data processing by running the programs stored in the memory 28, such as implementing the methods mentioned in the above embodiments.

[0134] In order to implement the above embodiments, the present disclosure further proposes a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method proposed in the above embodiments of the present disclosure is implemented.

[0135] In order to implement the above embodiments, the present disclosure further proposes a computer program product. When instructions in the computer program product are executed by a processor, the method proposed in the above embodiments of the present disclosure is executed.

[0136] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0137] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0138] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0139] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0140] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0141] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0142] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0143] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first scene image; Acquire monocular depth information and scene depth information associated with the first scene image; The first scene image is processed according to the monocular depth information and the scene depth information to obtain a target image.

2. The method according to claim 1, characterized in that The acquiring of monocular depth information and scene depth information related to the first scene image includes: Based on multiple camera devices, a plurality of frames of scene images are respectively collected, wherein the first scene image belongs to the plurality of frames of scene images; Determine monocular depth information corresponding to each frame of the scene image, and use the plurality of monocular depth information as monocular depth information associated with the first scene image; According to the multiple frames of scene images, scene depth information related to the first scene image is determined.

3. The method according to claim 2, characterized in that The step of processing the first scene image according to the monocular depth information and the scene depth information to obtain a target image includes: Determining a regional image corresponding to the scene image, wherein different regional images contain the same target object; The first scene image is processed according to the plurality of the region images, the monocular depth information, and the scene depth information to obtain the target image.

4. The method according to claim 3, characterized in that The determining of the regional image corresponding to the scene image comprises: Determining a candidate pixel feature for each candidate pixel in the scene image; Determining reference pixel features associated with the target object, wherein the reference pixel features include: at least one object feature of the target object mapped to a pixel feature of a pixel dimension; Processing the candidate pixel features and the reference pixel features to determine a probability value that the candidate pixel point belongs to the regional image; According to the plurality of probability values, a region image including the target object is detected from the scene image.

5. The method according to claim 4, characterized in that The detecting, from the scene image, a region image containing the target object according to the plurality of probability values ​​comprises: Determine, from the plurality of probability values, some probability values ​​that are greater than or equal to a probability threshold; The regional image is generated according to the candidate pixel points corresponding to each probability value in the partial probability values.

6. The method according to claim 2, characterized in that The determining of the monocular depth information corresponding to each frame of the scene image includes: The monocular depth information corresponding to each frame of the scene image is determined based on a monocular depth algorithm.

7. The method according to claim 6, characterized in that The determining the monocular depth information corresponding to each frame of the scene image based on the monocular depth algorithm includes: Inputting the scene image into a depth estimation model, wherein the depth estimation model is used to identify feature information from the scene image and predict a first scene depth of each pixel in the scene image based on the feature information; Acquire multiple first scene depths output by the depth estimation model, and use the multiple first scene depths as the monocular depth information.

8. The method according to claim 2, characterized in that The determining, according to the multiple frames of scene images, scene depth information related to the first scene image includes: Determine a scene disparity according to the multiple frames of scene images, wherein the scene disparity represents a horizontal offset of a captured scene on pixel planes of different camera devices; A second scene depth of the captured scene is determined according to the scene parallax, and the second scene depth is used as the scene depth information.

9. The method according to claim 1, characterized in that The number of the plurality of camera devices is two; Wherein, determining scene depth information related to the first scene image according to the multiple frames of scene images includes: The two frames of scene images are processed based on a dual-camera stereo matching algorithm to determine the scene depth information.

10. The method according to any one of claims 3 to 9, characterized in that: The step of processing the first scene image according to the plurality of the region images, the monocular depth information, and the scene depth information to obtain the target image includes: Generate a monocular depth image according to the monocular depth information, and generate a scene depth image according to the scene depth information; Determine, according to the plurality of area images, a monocular effective area image in the monocular depth image and a scene effective area image in the scene depth image; Acquire monocular feature information of the monocular effective area image and scene feature information of the scene effective area; According to the monocular feature information and the scene feature information, the monocular depth image and the scene depth image are fused to obtain a target depth image; The target image is synthesized according to the target depth image and the first scene image.

11. The method according to claim 10, characterized in that The monocular feature information and the scene feature information respectively include: color information and / or spatial domain information.

12. An image processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a first scene image; A second acquisition module, used to acquire monocular depth information and scene depth information related to the first scene image; A processing module is used to process the first scene image according to the monocular depth information and the scene depth information to obtain a target image.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images