Image processing method and apparatus

Abstract

Embodiments of the present application provide an image processing method and device, and relate to the technical field of terminals. The method comprises: obtaining a first disparity map and an edge region in a first image, the first disparity map being determined based on visual differences between the first image and a second image, and the edge region being a region constituted by a portrait feature in the first image; obtaining a first region with the same position as the edge region from the first disparity map, performing filtering processing on the first region to obtain a second region; fusing the second region into the first disparity map to obtain a second disparity map; and performing blurring processing on the first image based on the second disparity map to obtain a target image. In this way, since the portrait edge in the second disparity map is relatively clear, performing blurring processing on the first image based on the second disparity map can reduce the situation of edge false blurring or background missed blurring caused by disparity estimation errors, and improve the photographing quality.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and apparatus. Background Technology

[0002] With the popularization and development of the Internet, people's functional needs for electronic devices have become increasingly diversified. For example, electronic devices can not only support shooting functions, but also support bokeh processing of the captured images, making the images more vivid after bokeh processing.

[0003] Typically, to improve image quality, electronic devices can use binocular cameras to determine a parallax map, then estimate depth information, and based on the depth information, blur the captured image to obtain a blurred image.

[0004] However, the above image processing methods may result in false blurring or missed blurring in the blurred image. Summary of the Invention

[0005] This application provides an image processing method and apparatus, applied in the field of terminal technology, to improve the accuracy of image processing.

[0006] In a first aspect, embodiments of this application propose an image processing method. The method includes: in response to a photo-taking operation, acquiring a first image and a second image, wherein the first image and the second image are images acquired using a binocular camera; acquiring a first disparity map and edge regions in the first image, wherein the first disparity map is determined based on the visual differences between the first image and the second image, and the edge regions are regions composed of human features in the first image; acquiring a first region located at the same position as the edge regions from the first disparity map, filtering the first region to obtain a second region; fusing the second region into the first disparity map to obtain a second disparity map; and blurring the first image using the second disparity map to obtain a target image.

[0007] The edge region can be the first edge region described in the embodiments of this application, the first region can be the second edge region described in the embodiments of this application, and the second region can be the third edge region described in the embodiments of this application.

[0008] It is understandable that electronic devices can acquire a first image and a second image using a binocular camera. By semantically segmenting the first image, edge regions containing human figures are identified. A first region, located at the same position as the edge region, is obtained from the first disparity map. Filtering is then applied to this first region to improve the clarity of the human figures' edges. Finally, the first region and the first disparity map are fused to obtain a second disparity map. Since the human figures' edges are clearer in the second disparity map, blurring the first image based on the second disparity map can reduce false blurring of edges or background due to disparity estimation errors, thus improving image quality.

[0009] In one possible implementation, before obtaining the first disparity map and the edge regions in the first image, the method further includes: performing semantic segmentation on the first image to obtain a third image with human image regions; performing dilation on the third image to obtain a fourth image; and using the third image and the fourth image to determine the edge regions.

[0010] The third image can be the portrait segmentation map described in the embodiments of this application, and the fourth image can be the dilation map described in the embodiments of this application.

[0011] Electronic devices can perform dilation processing to ensure that the edge region encompasses all human image edges in the human image segmentation map. This allows subsequent camera algorithm libraries to obtain a first region from the first disparity map that corresponds to the edge region location, and then filter this first region to improve the accuracy of human image edges in the disparity map.

[0012] In one possible implementation, filtering the first region to obtain the second region includes: performing minimum filtering and / or Gaussian filtering on the first region to obtain the second region.

[0013] Since image segmentation maps cannot guarantee edge fit, electronic devices can use filtering to obtain more closely fitting edge regions. In the disparity map, the disparity of objects closer to the binocular cameras is greater. Therefore, minimum value filtering is used to obtain the minimum value of the disparity map within a certain region 1. This minimum value of the disparity map can be the disparity map of the background in region 1, in order to obtain clearer and more closely fitting image edges. Gaussian filtering can be used to smooth the image edges, making the edge transitions smoother.

[0014] In one possible implementation, obtaining the first disparity map includes: inputting a first image and a second image into a deep learning model, and outputting the first disparity map. The deep learning model is used to determine the first disparity map using portrait segmentation information of the first image, the first image, and the matching relationship between the first image and the second image.

[0015] Understandably, since the deep learning module can use human image segmentation information to optimize the generation of disparity maps, the first disparity map can have more accurate human image edges, thus improving the accuracy of the disparity map.

[0016] In one possible implementation, the deep learning model includes a correlation map calculation module and a disparity estimation module. The correlation map calculation module is used to perform feature matching on the first image and the second image to obtain a matching relationship. The disparity estimation module is used to determine the first disparity map using human image segmentation information and the matching relationship.

[0017] In one possible implementation, the deep learning model further includes: a hollow pyramid pooling (ASPP) module, a feature encoding module, and a feature decoding module; the feature encoding module is used to extract features from the first image and the second image respectively to obtain a first feature map corresponding to the first image and a second feature map corresponding to the second image; the ASPP module is used to extract features from the first feature map to obtain a third feature map; the feature decoding module is used to decode the third feature map into a fourth feature map, which contains human image segmentation information; and the correlation calculation module is used to perform feature matching between the first feature map and the second feature map to obtain a matching relationship.

[0018] like Figure 6 As described in the text, the first feature map can be a feature map. Figure 1 The second image can be a feature Figure 6 The third feature map can be a feature map. Figure 2 The fourth feature map can be a feature map. Figure 4 .

[0019] Understandably, the ASPP module can extract image features at any size through multiple dilated convolutions with different sampling rates, and the subsequent output of the ASPP module can yield more accurate semantic recognition results.

[0020] In one possible implementation, the deep learning model further includes: a first stitching module and a second stitching module. The method further includes: the first stitching module is used to stitch together the third feature map and the first feature map to obtain a fifth feature map; the second stitching module is used to stitch together the fourth feature map and the first feature map to obtain a sixth feature map, wherein the sixth feature map contains human image segmentation information.

[0021] like Figure 6 As described in the text, the first stitching module can be stitching module 603, and the second stitching module can be stitching module 605. The fifth feature map can be a feature map. Figure 3 The sixth feature map can be a feature map. Figure 5 .

[0022] It's understandable that each image processing step may result in the loss of some image information and the addition of others. Therefore, image stitching can increase the complexity of the image, thereby improving the accuracy of subsequent processing based on the stitched image.

[0023] Secondly, embodiments of this application provide an image processing method, including: acquiring a first image and a second image, wherein the first image and the second image are images acquired based on a binocular camera; inputting the first image and the second image into a deep learning model, and outputting a first disparity map, wherein the deep learning model is used to determine the first disparity map by utilizing the portrait segmentation information of the first image, the first image, and the matching relationship between the first image and the second image.

[0024] In one possible implementation, the deep learning model includes a correlation map calculation module and a disparity estimation module. The correlation map calculation module is used to perform feature matching on the first image and the second image to obtain a matching relationship. The disparity estimation module is used to determine the first disparity map using human image segmentation information and the matching relationship.

[0025] In one possible implementation, the deep learning model further includes: a hollow pyramid pooling (ASPP) module, a feature encoding module, and a feature decoding module; the feature encoding module is used to extract features from the first image and the second image respectively to obtain a first feature map corresponding to the first image and a second feature map corresponding to the second image; the ASPP module is used to extract features from the first feature map to obtain a third feature map; the feature decoding module is used to decode the third feature map into a fourth feature map, which contains human image segmentation information; and the correlation calculation module is used to perform feature matching on the first feature map and the second feature map to obtain a matching relationship.

[0026] In one possible implementation, the deep learning model further includes: a first stitching module and a second stitching module. The method further includes: the first stitching module is used to stitch together the third feature map and the first feature map to obtain a fifth feature map; the second stitching module is used to stitch together the fourth feature map and the first feature map to obtain a sixth feature map, wherein the sixth feature map contains human image segmentation information.

[0027] Thirdly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip or chip system within an electronic device. The image processing apparatus may include an acquisition unit and a processing unit. The acquisition unit is used to perform a data acquisition step to enable the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is an electronic device, the processing unit may be a processor. The image processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to enable the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to enable the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.), or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0028] Specifically, the image processing device includes an acquisition unit and a processing unit. In response to a photo-taking operation, the acquisition unit acquires a first image and a second image, both of which are images obtained using a binocular camera. The acquisition unit also acquires a first disparity map and edge regions within the first image. The first disparity map is determined based on the visual differences between the first and second images, and the edge regions are regions composed of human features in the first image. The processing unit acquires a first region located at the same position as the edge regions from the first disparity map, performs filtering on the first region to obtain a second region. The processing unit further merges the second region into the first disparity map to obtain a second disparity map. Finally, it performs a blurring process on the first image using the second disparity map to obtain a target image.

[0029] Fourthly, embodiments of this application provide an image processing apparatus, which may be an electronic device or a chip or chip system within an electronic device.

[0030] Specifically, the image processing device includes an acquisition unit and a processing unit. The acquisition unit is used to acquire a first image and a second image, which are images acquired based on a binocular camera. The processing unit is used to input the first image and the second image into a deep learning model and output a first disparity map. The deep learning model is used to determine the first disparity map using the portrait segmentation information of the first image and the matching relationship between the first image and the second image.

[0031] Fifthly, embodiments of this application provide an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect, or to perform the method described in the second aspect or any possible implementation of the second aspect.

[0032] Sixthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect or any possible implementation thereof, or to perform the method described in the second aspect or any possible implementation thereof.

[0033] In a seventh aspect, embodiments of this application provide a computer program product including a computer program. When the computer program product includes computer program code, when the computer program code is run on an electronic device, the electronic device performs the method described in the first aspect or any possible implementation of the first aspect, or performs the method described in the second aspect or any possible implementation of the second aspect.

[0034] Eighthly, this application provides a chip system applied to an electronic device. The chip system includes one or more processors, which are configured to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect or any possible implementation thereof, or to perform the methods described in the second aspect or any possible implementation thereof.

[0035] In one possible implementation, the chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip system, such as a register or cache, or it can be a storage unit of the chip system itself (e.g., read-only memory, random access memory, etc.).

[0036] It should be understood that the second to eighth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0037] Figure 1 A scenario diagram provided for an embodiment of this application;

[0038] Figure 2A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0039] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0040] Figure 4 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0041] Figure 5 A schematic image provided for an embodiment of this application;

[0042] Figure 6 A schematic diagram of a deep learning model provided in an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of the structure of a feature encoding module provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of the structure of an ASPP module provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of the structure of a feature decoding module provided in an embodiment of this application;

[0046] Figure 10 This is a schematic diagram of the structure of a portrait segmentation module provided in an embodiment of this application;

[0047] Figure 11 This application provides a schematic diagram of the steps of an image processing method according to an embodiment of the present application;

[0048] Figure 12 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0049] Figure 13 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0050] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0051] 1. Alpha mask (or simply mask)

[0052] A mask is an image generated by occluding (fully or partially) an image. Masks can be used to extract regions of interest. Taking an 8-bit mask data format as an example, the alpha values ​​of each pixel in the mask are illustrated below. In a mask, the alpha values ​​of pixels are distributed between 0 and 255. Foreground regions of interest can be set to alpha = 255, and background regions can be set to alpha = 0.

[0053] In this embodiment of the application, the electronic device can determine the human image region in the image by semantic segmentation of the image, and set the alpha of the human image region to 255 to obtain the human image segmentation map.

[0054] 2. Disparity Map

[0055] Disparity maps can be used to reflect the visual differences between two images acquired by a binocular camera. Binocular stereo vision fuses images obtained by both eyes and observes the differences between them to obtain a clear sense of depth. Electronic devices can establish correspondences between feature points to map the image points of the same spatial physical point in different images; the visual differences between feature points can form a disparity map.

[0056] It's understandable that the closer an object is to both eyes, the greater the parallax. For example, by placing a finger at different distances from the eyes and alternating between opening and closing the left and right eyes, it can be observed that the parallax varies depending on the distance of the finger, and the closer the distance, the greater the parallax.

[0057] There is a correlation between disparity and depth. After calculating the disparity map, electronic devices can convert it into a depth map.

[0058] 3. Atrous Spatial Pyramid Pooling (ASPP) Module

[0059] The ASPP module is designed for image segmentation tasks, addressing the issue of insufficient spatial context information in semantic segmentation. ASPP is based on the concept of dilated convolution. Dilated convolution is a technique that can increase the receptive field without increasing network parameters, helping the model capture a wider range of image information. The ASPP module employs multiple dilated convolutions; dilated convolutions with different sampling rates capture local information at different scales, ultimately resulting in feature maps with varying receptive fields.

[0060] The receptive field can be defined as the area in the input image that maps the pixels on the feature map output by each layer of the convolutional neural network back to the input image. Alternatively, it can be understood as the area of ​​the input image that the convolutional neural network's features can see relative to the size of the original image at a given point on the feature map.

[0061] 4. Other terms

[0062] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0063] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0064] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0065] 5. Electronic equipment

[0066] The electronic devices in this application embodiment may include handheld devices with display functions, vehicle-mounted devices, etc. For example, some electronic devices may be: mobile phones, tablet computers, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, etc., and this application embodiment is not limited to these.

[0067] By way of example and not limitation, in the embodiments of this application, the electronic device may also be a wearable device, such as glasses, gloves, watches, clothing and shoes.

[0068] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0069] In everyday photography, preserving the foreground's sharpness while blurring the background is a common technique. Electronic devices can improve image quality by blurring the background. During this process, electronic devices can rely on methods such as binocular stereo parallax or depth estimation to determine the subject and background areas, and then blur the background area to obtain the blurred result. However, if there are parallax estimation errors in the subject's area, the blurring result may result in the subject's edges being mistakenly blurred, or the background area being left unblurred.

[0070] The following is combined with Figure 1 The following is an illustrative description of the shooting scenario. For example, Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application. Figure 1 In the corresponding embodiments, a mobile phone is used as an example for illustration. This example does not constitute a limitation on the embodiments of this application.

[0071] like Figure 1 The scene shown in a can include a person in the foreground and a sun and sunlight in the background. At least one hair area can be displayed around the person's head, such as hair area 101 and hair area 102.

[0072] In response to a user opening the portrait mode in their camera, the electronic device can display something like... Figure 1The interface shown as b in the diagram can be the preview interface for a portrait photography function. This interface may include a shutter button 103 and a preview screen 104. The preview image displayed in the preview screen 104 can be obtained by the electronic device blurring the captured image. In possible implementations, the blurred preview image may not be displayed in the preview screen; this is not limited in this embodiment.

[0073] In response to the user's triggering operation on the camera button 103, the electronic device... Figure 1 Take a picture of the scene shown in Figure 'a' and obtain images such as... Figure 1 Image 105 is shown in the interface shown in 'c'. Image 105 can be obtained by blurring the captured image using an electronic device.

[0074] The captured image 105 may include: a clear hair area 101', a blurred hair area 102', and a clear partial background area 106, etc. (See also...) Figure 1 The scene in 'a' and Figure 1 In the photographed image shown in c, hair region 101 can be part of the foreground region, therefore hair region 101' can be a sharp area. Hair region 102 can be part of the foreground, but due to an error in parallax estimation by the electronic device, hair region 102 is mistakenly identified as the background, resulting in hair region 102' being blurred. The sun and its rays can be the background, but due to an error in parallax estimation by the electronic device, some sunlight rays are mistakenly identified as part of the portrait, resulting in some background region 106 not being blurred.

[0075] Understandably, parallax estimation errors can lead to situations where the edges of the portrait are mistakenly blurred or the background area is not blurred, affecting the quality of the photo.

[0076] In view of this, embodiments of this application provide an image processing method that enables an electronic device to acquire a first image and a second image based on a binocular camera. By semantically segmenting the first image, an edge region containing the human figure's edge is determined. A first region with the same location as the edge region is obtained from a first disparity map. The clarity of the human figure's edge in the first region is improved by filtering the first region. Then, the first region and the first disparity map are fused to obtain a second disparity map. Thus, since the human figure's edge is clearer in the second disparity map, blurring the first image based on the second disparity map can reduce the possibility of falsely blurred edges or missing backgrounds due to disparity estimation errors, thereby improving image quality.

[0077] Therefore, in order to better understand the embodiments of this application, the structure of the electronic device of the embodiments of this application will be described below. For example, Figure 2This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0078] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, an indicator 192, a camera 193, and a display screen 194, etc.

[0079] The sensor module 180 may include one or more of the following: a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, or a bone conduction sensor. Figure 2 (not shown in the text), etc., are not specifically limited in this embodiment.

[0080] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0081] The processor 110 may include one or more processing units. These processing units may be independent devices or integrated within one or more processors. The processor 110 may also include a memory for storing instructions and data. For example, the processor 110 is used to implement the steps performed in the image processing method provided in the embodiments of this application, and to store instructions and data related to the image processing method.

[0082] The charging management module 140 receives charging input from the charger. The charger can be a wireless charger or a wired charger. The power management module 141 connects the charging management module 140 to the processor 110.

[0083] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0084] Electronic devices utilize GPUs, displays (194), and application processors to achieve display functions. The GPU is a microprocessor for image processing, connecting the displays (194) and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. For example, the GPU is used to execute the graphics rendering process within an image processing method.

[0085] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. In some embodiments, an electronic device may include one or N displays screens 194, where N is a positive integer greater than 1. For example, display screen 194 is used to display preview images and captured images in a camera application.

[0086] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0087] Camera 193 is used to capture still images or videos. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1. For example, in response to a user's photo-taking operation, camera 193 may be used to acquire a sequence of raw images.

[0088] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The internal memory 121 can be used to store executable program code, including instructions. The internal memory 121 may include a program storage area and a data storage area. For example, the internal memory 121 can be used to store executable program code for an image processing method.

[0089] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0090] A touch sensor can be located on the display screen 194, and the touch sensor and the display screen 194 together form a touch screen, or "touchscreen". For example, the touch sensor is used to receive the user's trigger operation on the camera button.

[0091] Button 190 includes a power button, volume buttons, etc. Button 190 can be a mechanical button or a touch button. The electronic device can receive button input and generate key signal inputs related to user settings and function control of the electronic device. In some scenarios, the electronic device can also respond to the user's action on one or more buttons of button 190 to perform a photo-taking operation. The specific method of photo-taking operation is not limited in this embodiment.

[0092] The software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc., which will not be elaborated here.

[0093] Figure 3 This is a schematic diagram of the software structure of the electronic device provided in the embodiments of this application.

[0094] A layered architecture divides the system into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: application (APP), application framework layer (FWK), hardware abstraction layer (HAL), driver layer, and hardware layer.

[0095] The application layer may include a series of application packages. In this embodiment, the application package may include: a camera, a photo library, etc. The camera can capture images and display them. The photo library, also known as a photo album, can store and access captured images.

[0096] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment, the application framework layer may include a camera access interface, which may include camera management and camera devices. The camera access interface is used to provide application programming interfaces and programming frameworks for camera applications.

[0097] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0098] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. The camera algorithm library may include runtime code and data that implement the image processing methods provided in the embodiments of this application.

[0099] The driver layer is the layer between hardware and software. It includes drivers for various hardware components, such as camera drivers, digital signal processor drivers, and image processor drivers.

[0100] The camera device driver is used to drive the camera sensor to acquire images and to drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0101] The image processing method in this application embodiment will be described in detail below with reference to the above system structure:

[0102] In response to a user's action of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface in the application framework layer to launch the camera application. This then sends a command to the camera device (such as a stereo camera) in the camera hardware abstraction layer to start the camera. The camera hardware abstraction layer sends this command to the camera device driver in the kernel layer. The camera device driver can then start the corresponding camera sensor and acquire image light signals through the sensor. One camera device in the camera hardware abstraction layer corresponds to one camera sensor in the hardware layer.

[0103] Then, the camera sensor can transmit the acquired image light signal to the image signal processor for preprocessing to obtain the image electrical signal (raw image), and transmit the raw image to the camera hardware abstraction layer through the camera device driver. The raw image may include a first image and a second image.

[0104] The camera hardware abstraction layer can send the first image and the second image to a camera algorithm library. The camera algorithm library stores program code that implements the image processing method provided in the embodiments of this application. Based on a digital signal processor and an image processor, the camera algorithm library executes the above code to realize the process of generating a target image based on image processing of the first image and the second image.

[0105] The camera algorithm library can detect and send the raw images captured by the camera to the camera hardware abstraction layer. The camera hardware abstraction layer can then display these images.

[0106] It is understood that the software architecture provided in this application is only an example and does not constitute a limitation on the embodiments of this application.

[0107] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other. The same or similar concepts or processes may not be described again in some embodiments.

[0108] For example, Figure 4 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 4 In a corresponding embodiment, the electronic device may include: a gallery, a camera access interface, a camera algorithm library, and a camera device driver; the function of any of these modules can be found in [reference needed]. Figure 3 The corresponding implementation examples will not be described in detail here.

[0109] S401, In response to the photo-taking operation, the camera device driver acquires a first image and a second image.

[0110] The first image can be an image captured by the first camera, and the second image can be an image captured by the second camera.

[0111] Electronic devices may be equipped with binocular cameras, which may include a first camera and a second camera. When the first camera is the main camera in the binocular camera, the first image may be the image on the left; when the second camera is the auxiliary camera in the binocular camera, the second image may be the image on the right.

[0112] The response to the photo-taking operation includes: responding to the user's trigger operation on the photo-taking button in the portrait photo-taking function, or responding to the user's trigger operation on the photo-taking button in the HDR photo-taking function, or responding to the user's trigger operation on the photo-taking button in the large aperture photo-taking function, or responding to the user's trigger operation on the record button in the video recording function, etc. The applicable scenarios for the image processing method are not limited in the embodiments of this application.

[0113] S402, the camera algorithm library obtains the first image and the second image from the camera device driver.

[0114] The camera hardware abstraction layer can obtain the first image and the second image from the camera device driver, and the camera algorithm library can obtain the first image and the second image from the camera hardware abstraction layer.

[0115] S403, the camera algorithm library uses the first image and the second image to calculate the first disparity map and the portrait segmentation map.

[0116] The camera algorithm library can calculate the first disparity map and the portrait segmentation map based on the following two methods (including Method 1 and Method 2).

[0117] Method 1: The camera algorithm library can perform feature matching on the first and second images to obtain a correlation map, and then use the correlation map to calculate the first disparity map. The camera algorithm library can perform semantic segmentation on the first image and output a portrait segmentation map containing portrait edges.

[0118] The correlation map can be used to characterize the correlation between the first image and the second image. Correlation can be understood as the matching of feature points between two images. The feature matching method can include one or more of the following: based on speed-up robust features (SURF), scale-invariant feature transform (SIFT), or neural network models, etc.

[0119] A human face segmentation map can also be understood as a human face mask (or simply a mask map). In the mask map, the human face region can be a region with an alpha value of 255.

[0120] Method 2: Deep learning models can be pre-configured in the camera algorithm library. These models are used to output disparity maps and human segmentation maps based on the input binocular images.

[0121] For example, a camera algorithm library can input the first image and the second image into a trained deep learning model and output a first disparity map and a human segmentation map. See details. Figure 6 The description in the text will not be repeated here.

[0122] The deep learning model can perform semantic segmentation on the first image and calculate the corresponding portrait segmentation map. Furthermore, the deep learning model can also determine a second disparity map based on the first image, the second image, and the portrait segmentation map.

[0123] S404, the camera algorithm library dilates the portrait segmentation map and determines the first edge region based on the portrait segmentation map and the dilated map.

[0124] A dilation map can be understood as the image obtained after dilating a human face segmentation map. Dilation methods can include morphological filtering, etc.

[0125] The first edge region can be the image obtained by subtracting the dilation map and the portrait segmentation map from the camera algorithm library, and the first edge region can include the portrait edge.

[0126] Understandably, electronic devices can perform dilation processing to ensure that the first edge region encompasses all human image edges in the image segmentation map. This allows the subsequent camera algorithm library to obtain a second edge region located at the same position as the first edge region from the first disparity map, and then filter this second edge region to improve the accuracy of human image edges in the disparity map. The second edge region can be understood as the disparity region where the human image edges need optimization.

[0127] For example, Figure 5 This is a schematic diagram provided for an embodiment of this application. Figure 5 As shown, the portrait segmentation map can be Figure 5 The image shown in figure 'a' can be a dilatation map. Figure 5 In the image shown by b, the first edge region can be Figure 5 The black area shown as c in the diagram.

[0128] S405. The camera algorithm library uses the first edge region to obtain the second edge region from the first disparity map, performs filtering on the second edge region, and obtains the third edge region.

[0129] The meaning of the second edge region can be found in the description in S404, and will not be repeated here.

[0130] Filtering methods may include one or more of the following: minimum value filtering, or Gaussian filtering, etc.

[0131] Understandably, since the image segmentation map cannot guarantee edge fit, electronic devices can use filtering to obtain a more closely fitting edge region. In the disparity map, the disparity of objects closer to the binocular camera is greater. Therefore, minimum value filtering is used to obtain the minimum value of the disparity map within a certain region 1. This minimum value of the disparity map can be the disparity map of the background in region 1, in order to obtain a clearer and more closely fitting image edge. Gaussian filtering can be used to smooth the image edges, making the edge region transition more smoothly.

[0132] In this way, by applying minimum filtering and Gaussian filtering to the second edge region, the edges of the person in the second edge region are made closer to those in the first edge region. After filtering, the edges of the person in the third edge region are sharper, providing more accurate parallax information for subsequent blurring effects.

[0133] S406, the camera algorithm library performs image fusion on the first disparity map and the third edge region to obtain the second disparity map.

[0134] For example, a camera algorithm library can perform image fusion on the regions other than the third edge region in the first disparity map, as well as the third edge region, to obtain a second disparity map with optimized portrait edges, thereby improving the accuracy of the second disparity map.

[0135] S407. The camera algorithm library uses the second disparity map to blur the first image to obtain the target image.

[0136] Blur processing can be achieved through methods such as Gaussian blurring, and this application does not limit the specific methods used in the embodiments.

[0137] S408: The image library obtains the target image from the camera algorithm library.

[0138] For example, the camera algorithm library can return the processed target image to the image library through modules such as the camera access interface.

[0139] S409. The image library stores the target image.

[0140] Understandable Figure 4 The portrait segmentation map described herein can not only be used to enhance the edges of portraits in parallax maps, but also applied to processing methods such as skin smoothing, portrait enhancement, or image cutout. The application scenarios of the portrait segmentation map are not limited in this application embodiment.

[0141] Based on this, electronic devices can obtain a first edge region with human image edges through semantic segmentation and dilation processing, and then obtain a second edge region matching the first edge region from a first disparity map. Filtering of the second edge region yields a second disparity map with clearer human image edges. After blurring the first image based on the second disparity map, a target image with accurate human image edge blurring can be obtained, reducing the possibility of falsely blurred edges or missing background blur due to disparity estimation errors.

[0142] Based on the description in S403, the camera algorithm library is based on a deep learning model. The process of outputting the first disparity map and the human image segmentation map can be found below. Figure 6 The description in the document states that the deep learning model can be a gated recurrent unit (GRU) iterative model or other models.

[0143] For example, Figure 6 This is a schematic diagram of a deep learning model provided in an embodiment of this application. Figure 6 As shown, the deep learning model may involve one or more of the following: left image feature encoding module 601, ASPP module 602, stitching module 603, left image feature decoding module 604, stitching module 605, human image segmentation module 606, right image feature encoding module 607, correlation map calculation module 608, or disparity estimation module 609.

[0144] The feature encoding module 601 in the left image is used to output the feature map corresponding to the first image, and the feature encoding module 607 in the right image is used to output the feature map corresponding to the second image. The feature encoding module 601 in the left image and the feature encoding module 607 in the right image can be feature encoding modules with the same structure; the structure of the feature encoding module can be found in [reference needed]. Figure 7 The description in the text will not be repeated here.

[0145] ASPP module 602 is used for features; the structure of the ASPP module can be found in [reference needed]. Figure 8 The description in the text will not be repeated here.

[0146] Both stitching modules 603 and 605 are used for image stitching. It is understandable that each image processing step may result in the loss of some image information and the addition of some information. Therefore, image stitching can increase the complexity of the image, thereby improving the accuracy of subsequent processing based on the stitched image.

[0147] The feature decoding module 604 in the left image is used to decode the feature map and output a feature map containing human face segmentation information. The structure of the feature decoding module (or feature decoding module) 604 in the left image can be found in [reference needed]. Figure 9The description in the text will not be repeated here.

[0148] The portrait segmentation module 606 can be understood as a decoding module, that is, it decodes the feature map into a portrait segmentation map that the user can recognize. The structure of the portrait segmentation module can be found in [reference needed]. Figure 10 The description in the text will not be repeated here.

[0149] Among them, ASPP module 602, stitching module 603, left image feature decoding module 604, stitching module 605, and portrait segmentation module 606 can work together to achieve semantic segmentation of the image (such as the area shown in the dashed box).

[0150] The correlation map calculation module 608 is used to perform feature matching on the first image and the second image to obtain the matching relationship between feature points. In the deep learning model, the correlation map calculation module can generate a correlation feature map based on the input feature map, which can be used to represent the feature matching relationship between feature maps.

[0151] The disparity estimation module 609 is used to calculate a disparity map based on the correlation feature map, the portrait segmentation map, and the first image. The first image can be a reference image, allowing the first disparity map output by the electronic device based on the disparity estimation module 609 to be closer to the first image. The portrait segmentation map contains portrait segmentation information, which characterizes portrait edges and assists in estimating the first disparity map, enabling it to accurately identify portrait edges.

[0152] For example, when feature 1 in the correlation feature map is determined to be the background, and feature 1 in the portrait segmentation map is the portrait, the electronic device can use the portrait segmentation map as a reference to determine feature 1 as the portrait, and then generate a first disparity map with accurate portrait edges.

[0153] like Figure 6 As shown, the process of obtaining the portrait segmentation map corresponding to the first image may include the following steps. The camera algorithm library can input the first image into the feature encoding module 601 in the left image to encode the features corresponding to the first image. Figure 1 The camera algorithm library will use features Figure 1 Input to ASPP module, output characteristics Figure 2 The camera algorithm library will use features Figure 1 and features Figure 2 The input is fed into the splicing module 603, and the features are spliced ​​together. Figure 3 The camera algorithm library will use features Figure 3 The input is fed into the feature decoding module 604 in the left image, and the features are decoded to obtain the features. Figure 4 The camera algorithm library will use features Figure 1 and features Figure 4The input is fed into the splicing module 605, and the features are spliced ​​together. Figure 5 The camera algorithm library will use features Figure 5 The input is fed into the portrait segmentation module, which outputs a portrait segmentation map that the user can recognize. Among these features... Figure 4 and features Figure 5 Each of these can contain human image segmentation information.

[0154] The process of obtaining a first disparity map based on the first image and the second image may include the following steps. The camera algorithm library inputs the second image into the feature encoding module 607 shown in the right figure to obtain features. Figure 6 The camera algorithm library will use features Figure 6 and features Figure 1 The input is fed into the correlation map calculation module 608, which outputs a correlation feature map. The camera algorithm library combines the correlation feature map and features... Figure 1 and characteristics Figure 5 The input is fed into the disparity estimation module 609, which outputs the first disparity map.

[0155] Understandably, deep learning models can determine a first disparity map based on the matching relationship between the first and second images, portrait segmentation information, and the first image, thus enabling the identification of relatively accurate portrait edges in the first disparity map.

[0156] For example, Figure 7 This is a schematic diagram of the structure of a feature encoding module provided in an embodiment of this application.

[0157] like Figure 7 As shown, the feature decoding module can consist of 4 blocks. Each block can be composed of: convolution (conv) + instance normalization (IN) + activation function (relu), as shown in the dashed box. The stride in block 1 and block 3 can be 2, and the stride in block 2 and block 4 can be 1.

[0158] The camera algorithm library can also calculate the residual between the output of block 1 and the output of block 2, the residual between the output of block 2 and the output of block 3, and the residual between the output of block 3 and the output of block 4, so as to better implement gradient descent through residual analysis.

[0159] For example, the camera algorithm library can input an image of size 1×3×H×W sequentially into 4 blocks, and output an image of size 1×C×H / 4×W / 4. The input image can be... Figure 6 The first or second image described in the text. Camera algorithm libraries can be used... Figure 7 The three convolutions in the image yield an output image with a total of C channels and 1 / 4 of the original size.

[0160] Understandable Figure 7 The structure of the feature encoding module provided is only an example and does not constitute a limitation on the embodiments of this application.

[0161] For example, Figure 8 This is a schematic diagram of the structure of an ASPP module provided in an embodiment of this application.

[0162] like Figure 8 As shown, the ASPP module can consist of multiple dilated convolutions with different sampling rates, such as a conv1×1 dilated convolution with a sampling rate of 0, a conv3×3 dilated convolution with a sampling rate of 1, a conv3×3 dilated convolution with a sampling rate of 5, and a conv3×3 dilated convolution with a sampling rate of 9. These dilated convolutions of different sizes can be used to extract feature maps of different sizes. The ASPP module can also include steps such as pooling and upsampling.

[0163] For example, a camera algorithm library can input an image into multiple dilated convolutions to obtain multiple output results. These multiple output results are then concatenated and subjected to a conv1×1 convolution to obtain the output image.

[0164] Typically, to fix the size of the input image, electronic devices can perform image processing such as cropping or stretching. However, these processing methods result in a loss of image precision, affecting the accuracy of subsequent semantic recognition results. In this embodiment, the ASPP module can extract image features at any size through multiple dilated convolutions with different sampling rates. The subsequent semantic recognition results based on the output of the ASPP module can obtain more accurate semantic recognition results.

[0165] Understandable Figure 8 The structure of the ASPP module provided is only an example and does not constitute a limitation on the embodiments of this application.

[0166] For example, Figure 9 This is a schematic diagram of a feature decoding module provided in an embodiment of this application. The feature decoding module and the feature encoding module do not necessarily need to correspond one-to-one.

[0167] like Figure 9 As shown, the feature decoding module can be composed of multiple blocks, such as a conv3×3 convolutional block with a stride of 1, a conv1×1 convolutional block with a stride of 1, and a bilinear upsampling module, etc.

[0168] For example, the camera algorithm library can sequentially input an image of size 1×C×H / 4×W / 4 into the above convolutional block and bilinear upsampling module, and output an image of size 1×C×H / 2×W / 2.

[0169] Understandable Figure 9 The structure of the feature encoding module provided is only an example and does not constitute a limitation on the embodiments of this application.

[0170] For example, Figure 10 This is a schematic diagram of the structure of a portrait segmentation module provided in an embodiment of this application.

[0171] like Figure 10 As shown, the feature decoding module can be composed of multiple blocks, such as a conv3×3 convolutional block with a group convolution value of C, a conv1×1 convolutional block with a group convolution value of 1, a conv3×3 convolutional block with a group convolution value of 1, and a bilinear upsampling module, etc.

[0172] For example, the camera algorithm library can sequentially input an image of size 1×C×H / 2×W / 2 into the above convolutional block and bilinear upsampling module, and output a portrait segmentation map of size 1×3×H×W.

[0173] Understandable Figure 10 The structure of the human image segmentation module provided is only an example and does not constitute a limitation on the embodiments of this application.

[0174] In combination with the above Figures 4-10 The description in the document is for the purpose of more clearly illustrating the embodiments of this application. Figure 11 This is a schematic diagram illustrating the steps of an image processing method provided in an embodiment of this application. For example... Figure 11 As shown, the image processing method may include the following steps:

[0175] S1101. The electronic device acquires a human image segmentation image and dilates the human image segmentation image to obtain an inflated image.

[0176] The human image segmentation map can be determined based on the first image. The process by which the electronic device determines the human image segmentation map based on the first image can be found in the description in S403, and will not be repeated here.

[0177] S1102, The electronic device determines the edge region of the human image based on the human image segmentation map and the dilation map.

[0178] The edge area of ​​the portrait can be Figure 4 The first edge region described in the figure can be obtained by subtracting the dilation map from the figure segmentation map.

[0179] S1103. The electronic device obtains the image edge region to be optimized from the first disparity map, which is at the same position as the image edge region.

[0180] The edge area of ​​the portrait to be optimized can be Figure 4 The second edge region described in the text.

[0181] S1104. The electronic device performs minimum value filtering on the edge region of the portrait to be optimized.

[0182] S1105. The electronic device performs Gaussian filtering on the edge region of the portrait to be optimized.

[0183] The effects of minimum filtering and Gaussian filtering can be found in the description in S405, and will not be repeated here.

[0184] After minimum filtering and Gaussian filtering, the electronic device can obtain the optimized portrait edge region. The optimized portrait edge region can be... Figure 4 The third edge region described in the image. Further, the electronic device can perform image fusion on the third edge region and the first disparity map to obtain a second disparity map, and then perform blurring processing on the first image based on the second disparity map to obtain the target image.

[0185] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0186] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0187] The image processing method of the present application embodiments has been described above. The apparatus for performing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the present application embodiments can perform the steps in the above list sorting method.

[0188] like Figure 12 As shown, Figure 12 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. The image processing device may be an electronic device in the embodiment of this application, or it may be a chip or chip system within an electronic device.

[0189] like Figure 12 As shown, the image processing apparatus 1200 can be used in communication devices, circuits, hardware components, or chips. The image processing apparatus 1200 includes an acquisition unit 1201 and a processing unit 1202. The acquisition unit 1201 supports the data acquisition step of the image processing method; the processing unit 1202 supports the information processing step of the image processing apparatus 1200.

[0190] In one possible implementation, the image processing apparatus 1200 may further include a communication unit 1203, which supports the image processing apparatus 1200 in performing steps such as receiving or sending messages.

[0191] The image processing apparatus described in the embodiments of this application may include all of the following: Figure 12 The units described in the corresponding embodiments.

[0192] Specifically, the processing unit 1202 and the acquisition unit 1201 can be integrated together, and the processing unit 1202 and the acquisition unit 1201 may communicate with each other.

[0193] In one possible implementation, the image processing apparatus 1200 may further include a storage unit 1204. The storage unit 1204 may include one or more memories, which may be devices in one or more devices or circuits used for storing programs or data.

[0194] The storage unit 1204 can exist independently and be connected to the processing unit 1202 via a communication bus. Alternatively, the storage unit 1204 can be integrated with the processing unit 1202.

[0195] Taking the image processing device 1200 as an example, which may be a chip or chip system of the electronic device in the embodiments of this application, the storage unit 1204 may store computer-executable instructions of the electronic device method, so that the processing unit 1202 executes the electronic device method in the above embodiments. The storage unit 1204 may be a register, cache, or random access memory (RAM), etc., and the storage unit 1204 may be integrated with the processing unit 1202. The storage unit 1204 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 1204 may be independent of the processing unit 1202.

[0196] In one possible implementation, the image processing apparatus 1200 may further include a communication unit 1203. The communication unit 1203 supports interaction between the image processing apparatus 1200 and other devices. For example, when the image processing apparatus 1200 is an electronic device, the communication unit 1203 may be a communication interface or interface circuit. When the image processing apparatus 1200 is a chip or chip system within an electronic device, the communication unit 1203 may be a communication interface. For example, a communication interface may be an input / output interface, pins, or circuits.

[0197] The apparatus in this embodiment can be used to execute the steps performed in the above method embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0198] Figure 13 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application.

[0199] The electronic device includes a processor 1301, a communication line 1304, and at least one communication interface. Figure 13 (The example described uses communication interface 1303 as an example).

[0200] The processor 1301 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0201] Communication line 1304 may include circuitry for transmitting information between the aforementioned components.

[0202] Communication interface 1303 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0203] Possibly, the electronic device may also include a memory 1302.

[0204] The memory 1302 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via communication line 1304. The memory may also be integrated with the processor.

[0205] The memory 1302 stores computer execution instructions for implementing the scheme of this application, and the processor 1301 controls the execution. The processor 1301 executes the computer execution instructions stored in the memory 1302 to implement the method provided in the embodiments of this application.

[0206] It is possible that the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0207] In a specific implementation, as one example, the processor 1301 may include one or more CPUs, for example... Figure 13 CPU0 and CPU1 in the CPU.

[0208] In a specific implementation, as one example, an electronic device may include multiple processors, for example... Figure 13 Processors 1301 and 1305 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0209] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0210] The image processing method provided in this application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can be referred to the above-described related descriptions, which will not be repeated here.

[0211] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.

[0212] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0213] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0214] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0215] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0216] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0217] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: In response to a photo-taking operation, a first image and a second image are acquired, wherein the first image and the second image are images acquired based on a binocular camera; The first image and the second image are input into a deep learning model, which then outputs a first disparity map and a portrait segmentation map simultaneously. The image segmentation map is dilated to obtain an inflated map; Based on the portrait segmentation map and the dilation map, a first edge region is determined, which is a strip-shaped region formed by the portrait edges in the first image; A second edge region with the same position as the first edge region is obtained from the first disparity map. The second edge region is then subjected to minimum value filtering and / or Gaussian filtering to obtain a third edge region. Image fusion is performed on the first disparity map and the third edge region to obtain a second disparity map; The target image is obtained by blurring the first image using the second disparity map. The deep learning model includes: a left-image feature encoding module, used to extract features from the first image to obtain a first feature map corresponding to the first image; The hollow pyramid pooling ASPP module is used to extract features from the first feature map to obtain the third feature map; The first splicing module is used to splice the third feature map and the first feature map to obtain the fifth feature map; The feature decoding module is used to decode the fifth feature map into a fourth feature map, wherein the fourth feature map contains human image segmentation information; The second stitching module is used to stitch the fourth feature map and the first feature map to obtain a sixth feature map, which contains the human image segmentation information. The portrait segmentation module is used to output the portrait segmentation map based on the sixth feature map; The feature encoding module in the right figure is used to extract features from the second image to obtain the second feature map corresponding to the second image; The correlation graph calculation module is used to perform feature matching on the first feature map and the second feature map to obtain a correlation feature map, which is used to characterize the feature matching relationship between the feature maps. The disparity estimation module is used to output the first disparity map based on the correlation feature map, the first feature map, and the sixth feature map.

2. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in claim 1.

3. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in claim 1.

5. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in claim 1.

Images