Image processing method and apparatus

Abstract

The embodiment of the application provides an image processing method and device, relates to the terminal technical field, so that the electronic device can acquire a mask image corresponding to a clear image and a ternary image corresponding to the clear image, the position of an edge feature is determined through the mask image, a transition region including the edge feature is determined through the ternary image, and then the electronic device can acquire an edge region corresponding to the position of the edge feature from the clear image based on the position of the edge feature, and fuse the edge region into a blurred image, so that the situation that an edge is wrongly blurred or a background is wrongly clear in the blurred image is solved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and apparatus. Background Technology

[0002] With the popularization and development of the Internet, people's functional needs for electronic devices have become increasingly diversified. For example, electronic devices can not only support shooting functions, but also support the blurring of the captured image, allowing users to see a blurred image with a clear foreground and a blurred background, giving the captured image a better sense of space.

[0003] However, due to limitations in the hardware modules of electronic devices, blurred images may exhibit issues such as blurred edges or unclear backgrounds. Summary of the Invention

[0004] This application provides an image processing method and apparatus for improving the blurring effect on edge features.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising: in response to a photo-taking operation, acquiring a first image; obtaining a blurred image corresponding to the first image, the positions of edge features in the first image, and a transition region in the first image composed of the edge features; wherein the blurred image is obtained by blurring the background in the first image; using the positions of the edge features, determining a first region composed of the edge features from the first image, and fusing the first region into the blurred image to obtain a target image; wherein the first region is located within the transition region.

[0006] Electronic devices can accurately filter out a first region composed of clear edge features from a first image by using the location of edge features and the first region, and solve the problem of falsely blurred edges or falsely clear backgrounds in the blurred image by fusing the first region into the blurred image.

[0007] In one possible implementation, before obtaining the position of the edge features in the first image and the transition region formed by the edge features in the first image, the method further includes: obtaining a mask image corresponding to the first image and a ternary image corresponding to the first image, wherein the mask image corresponding to the first image includes the position of the edge features and the ternary image corresponding to the first image includes the transition region.

[0008] The mask image corresponding to the first image can be the first mask image described in the embodiments of this application, and the ternary image corresponding to the first image can be the first ternary image described in the embodiments of this application.

[0009] This allows electronic devices to acquire a mask image and a ternary image corresponding to a clear image. The mask image is used to determine the position of edge features, and the ternary image is used to determine the transition region including the edge features. Then, the electronic device can obtain the edge region corresponding to the position of the edge features from the clear image based on the position of the edge features, and fuse the edge region into the blurred image, thus solving the problem of falsely blurred edges or falsely clear backgrounds in blurred images.

[0010] In one possible implementation, obtaining the mask image and the ternary image corresponding to the first image includes: inputting the first image into a first model and outputting the mask image and the ternary image corresponding to the first image; wherein the first model includes: a shared editor, a semantic segmentation branch, a detail prediction branch, and a fusion branch, the shared editor being used to extract image features of the first image, the semantic segmentation branch being used to perform semantic segmentation on the first image and obtain the foreground in the first image, the detail prediction branch being used to obtain the edge features in the first image, and the fusion branch being used to fuse the foreground and the edge features in the first image to obtain the mask image and the ternary image corresponding to the first image.

[0011] Electronic devices can output a mask that can identify clear edges and a ternary map that can frame edge features within transition regions through a target neural network model. This allows the precise location and extent of edge features to be determined using the mask and the ternary map.

[0012] In one possible implementation, the method further includes: the first model is trained by: inputting a training image into the shared editor to obtain image features of the training image; inputting the image features of the training image into the semantic segmentation branch to obtain the output result of the semantic segmentation branch; obtaining the Laplacian high-frequency map corresponding to the training image, inputting the Laplacian high-frequency map into the detail prediction branch to obtain the output result of the detail prediction branch; inputting the output result of the semantic segmentation branch and the output result of the detail prediction branch into the fusion branch to obtain a predicted mask map and a predicted ternary map; and obtaining the trained first model when the difference between the predicted mask map and the ground truth of the mask map satisfies a first loss function and the difference between the predicted ternary map and the ground truth of the ternary map satisfies a second loss function.

[0013] The training device can identify the foreground contour of an image through the semantic segmentation branch output, determine the range of edge features based on the detail prediction branch, and obtain a clear contour and a region containing edge features through the fusion branch.

[0014] In one possible implementation, the method further includes: performing Gaussian blurring on the edges of the ground truth of the mask image to obtain a second image; and obtaining the trained semantic segmentation branch when the difference between the second image and the output of the semantic segmentation branch satisfies a fourth loss function.

[0015] The second image can be the mask edge blur map described in the embodiments of this application, and the fourth loss function can be the loss4 described in the embodiments of this application.

[0016] Since the semantic segmentation branch performs relatively coarse semantic segmentation of the image and can only obtain a rough foreground of the human figure, while the edge of the ground truth mask is more accurate, in order for the semantic segmentation branch to output an image that is consistent with the foreground contour in the ground truth mask, the training device can perform loss4 calculation on the ground truth mask and the output of the semantic segmentation branch.

[0017] In one possible implementation, the method further includes: obtaining an image within the transition region from the mask image ground truth according to the transition region in the ternary image ground truth, to obtain a third image; and obtaining the trained detail prediction branch when the difference between the third image and the output of the detail prediction branch satisfies a third loss function.

[0018] The third image is the transition region mask described in the embodiments of this application, and the third loss function can be loss3 described in the embodiments of this application.

[0019] To enable the detail prediction branch to predict more details within transition regions that include edge features, the training device can perform loss3 calculations on the ground truth of the ternary map and the output of the detail prediction branch.

[0020] In one possible implementation, the method further includes: obtaining a second region containing edge features from the training image based on a semantic segmentation method; performing connected component filtering on the second region to obtain a third region; dilating the third region to obtain a fourth region; dilating the ground truth of the mask image to obtain a dilated mask image; obtaining the intersection of the dilated mask image and the fourth region to obtain a first transition region; and determining the ground truth of the ternary image based on the first transition region and the foreground in the dilated mask image.

[0021] Electronic devices can determine the location of edge features by filtering connected components and determine the regions containing edge features by operations such as dilation, so as to obtain a more accurate true value of the ternary graph.

[0022] Furthermore, compared to the conventional method of directly dilating the mask image to obtain a ternary map where the transition region encompasses the entire image, the transition region of the ground truth ternary map in this application only contains edge feature distributions.

[0023] In one possible implementation, the edge features include one or more of the following: human hair strands, yarn in a sweater, animal hair, or plant branches and leaves.

[0024] In one possible implementation, the step of acquiring the first image in response to the photo-taking operation includes: acquiring at least two images with different exposure times in response to the photo-taking operation, and performing image fusion on the at least two images with different exposure times to obtain the first image.

[0025] Electronic devices can obtain a first image with HDR effect by image fusion between at least two images with different exposure times.

[0026] Secondly, embodiments of this application provide an image processing method, the method comprising: a first model being trained by: inputting a training image into the shared editor to obtain image features of the training image; inputting the image features of the training image into the semantic segmentation branch to obtain the output result of the semantic segmentation branch; obtaining the Laplacian high-frequency map corresponding to the training image, inputting the Laplacian high-frequency map into the detail prediction branch to obtain the output result of the detail prediction branch; inputting the output result of the semantic segmentation branch and the output result of the detail prediction branch into the fusion branch to obtain a predicted mask map and a predicted ternary map; and obtaining the trained first model when the difference between the predicted mask map and the ground truth of the mask map satisfies a first loss function and the difference between the predicted ternary map and the ground truth of the ternary map satisfies a second loss function.

[0027] The subject of training the first model can be either an electronic device or a server; this embodiment of the application does not limit this.

[0028] In one possible implementation, the method further includes: performing Gaussian blurring on the edges of the ground truth of the mask image to obtain a second image; and obtaining the trained semantic segmentation branch when the difference between the second image and the output of the semantic segmentation branch satisfies a fourth loss function.

[0029] In one possible implementation, the method further includes: obtaining an image within the transition region from the mask image ground truth according to the transition region in the ternary image ground truth, to obtain a third image; and obtaining the trained detail prediction branch when the difference between the third image and the output of the detail prediction branch satisfies a third loss function.

[0030] In one possible implementation, the method further includes: obtaining a second region containing edge features from the training image based on a semantic segmentation method; performing connected component filtering on the second region to obtain a third region; dilating the third region to obtain a fourth region; dilating the ground truth of the mask image to obtain a dilated mask image; obtaining the intersection of the dilated mask image and the fourth region to obtain a first transition region; and determining the ground truth of the ternary image based on the first transition region and the foreground in the dilated mask image.

[0031] In one possible implementation, the edge features include one or more of the following: human hair strands, yarn in a sweater, animal hair, or plant branches and leaves.

[0032] Thirdly, embodiments of this application provide an image processing apparatus, which includes a display unit and a processing unit. The display unit is used to process the step of displaying data in the image processing apparatus, and the processing unit is used to process the step of data processing in the image processing apparatus.

[0033] In one possible implementation, the image processing apparatus may further include a storage unit, which may include one or more memories, which may be devices in one or more devices or circuits used to store programs or data.

[0034] Fourthly, embodiments of this application provide an electronic device including a processor and a memory, the memory being used to store code instructions; the processor being used to execute the code instructions, causing the electronic device to perform the method described in the first aspect or any implementation thereof.

[0035] Fifthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed, cause a computer to perform the method described in the first aspect or any implementation thereof.

[0036] A sixth aspect is a computer program product comprising a computer program that, when run, causes a computer to perform the methods described in the first aspect or any implementation thereof.

[0037] A seventh aspect, a chip, comprising: a processor for reading instructions stored in a storage memory, wherein when the processor executes the instructions, the chip causes the chip to perform the method described in the first aspect or any implementation thereof.

[0038] It should be understood that the second to seventh aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0039] Figure 1 A scenario diagram provided for an embodiment of this application;

[0040] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of the software structure of the electronic device provided in the embodiments of this application;

[0042] Figure 4 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0043] Figure 5 A schematic diagram of the architecture of a neural network model provided in an embodiment of this application;

[0044] Figure 6 A schematic image provided for an embodiment of this application;

[0045] Figure 7 This application provides a schematic diagram of the steps for generating a ternary graph according to an embodiment of the present application;

[0046] Figure 8 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0048] Figure 10 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0049] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms used in the embodiments of this application are explained. It should be understood that this explanation is for the purpose of more clearly explaining the embodiments of this application and does not necessarily constitute a limitation on the embodiments of this application.

[0050] 1. Previewing and capturing images

[0051] Preview images can be data captured in real-time by the camera of an electronic device and displayed in the preview screen. For example, when an electronic device receives a user's command to open the camera application, it can capture a preview image from the camera and display it in real-time in the camera application's preview screen.

[0052] The captured image can be data obtained based on the camera button in the electronic device, such as the target image described in the embodiments of this application. For example, when the electronic device receives a user's trigger operation on the camera button, the electronic device can acquire the captured image based on the camera at the moment of taking the photo.

[0053] 2. High Dynamic Range (HDR)

[0054] HDR is a processing technique that enhances the brightness and contrast of images. Compared to ordinary images, HDR can provide more dynamic range and image detail. It uses images with the best detail corresponding to each exposure time to synthesize the final HDR image, which can better reflect the visual effects of the real environment.

[0055] One possible implementation for an electronic device to determine whether the current scene is an HDR scene is as follows: The electronic device downsamples the preview image by 4 times to obtain a preview thumbnail, and determines whether the proportion of the number of bright pixels in the preview thumbnail to the total number of pixels in the preview thumbnail is greater than a preset pixel threshold. The preview thumbnail can be obtained by storing every row of pixels in the frame corresponding to the preview image, keeping one row of pixels for every two rows. The bright pixels can be determined based on a grayscale threshold, which can be used to determine whether the current scene is a high dynamic range scene. The electronic device can determine whether the current scene is a high dynamic range scene based on one frame of data or multiple frames of data; this embodiment does not limit this.

[0056] 3. Camera depth of field

[0057] Depth of field can be understood as the imaging range within a camera lens or other imager that yields a sharp image, or as the range of sharpness before and after the focus point. The focus point can be the point of sharpness obtained when light is focused onto the image sensor through the lens. Foreground depth of field includes the range of sharpness before the focus point, while background (or rearfield) depth of field includes the range of sharpness after the focus point.

[0058] Key factors affecting depth of field include aperture, lens, and distance from the subject. A larger aperture (smaller f-number) results in a shallower depth of field, while a smaller aperture (larger f-number) results in a deeper depth of field. Similarly, a longer focal length results in a shallower depth of field, while a shorter focal length results in a deeper depth of field.

[0059] 4. Exposure time (or exposure duration)

[0060] Exposure time is the time the shutter needs to be open to project light onto the photosensitive surface of a photographic material, or it can be understood as the time interval between the shutter opening and closing.

[0061] Exposure time refers to the exposure time of the film. The longer the exposure time, the brighter the resulting photograph, and vice versa. In low-light conditions, it is generally necessary to extend the exposure time to obtain a brighter image.

[0062] 5. Alpha mask (or simply mask)

[0063] A mask is an image generated by occluding (fully or partially) an image. Masks can be used to extract regions of interest. Taking an 8-bit mask data format as an example, the alpha values ​​of each pixel in the mask are illustrated below. In a mask, the alpha values ​​of pixels are distributed between 0 and 255. Foreground regions of interest can be set to alpha = 255, and background regions can be set to alpha = 0.

[0064] Typically, image matting can be achieved using mask images. For example, the image matting problem can be modeled as follows:

[0065] I = alpha × Fg + (255 - alpha) × Bg

[0066] Where I represents the complete image, Fg represents the foreground of the portrait, Bg represents the background, and alpha represents the mask of the foreground Fg. The value of alpha ranges from 0 to 255. Regions with an alpha value of 0 can be considered background regions, regions with an alpha value of 255 can be considered foreground regions, and regions with an alpha value greater than 0 and less than 255 are transition regions in the mask image.

[0067] 6. Trimap

[0068] A trimap can be an image containing three types of markers: foreground, background, and a blended foreground / background region. Each pixel in a trimap can have one of three regions: 0, 128, or 255. Trimaps can provide a coarse division of a given image. For example, a region with trimap = 0 can be considered the background region, a region with trimap = 255 can be considered the foreground region, and a region with trimap = 128 can be considered the transition region.

[0069] 7. Electronic equipment

[0070] Electronic devices can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices can include mobile phones with touchscreens, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments in this application do not limit the specific technologies or device forms used in the electronic devices.

[0071] 8. Other

[0072] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. For example, the first value and the second value are only used to distinguish different values ​​and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0073] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0074] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0075] In everyday photography, preserving the foreground's sharpness while blurring the background is a common technique. However, due to hardware limitations, mobile phone cameras often fail to achieve sufficient background blur when capturing images directly. Algorithms are needed to extract the foreground from the image before blurring the background. Therefore, the accuracy of edge feature extraction significantly impacts the final blurred image. However, because edge features are complex and detailed, blurred images are prone to errors such as inaccurate edge blurring or inaccurate background sharpness.

[0076] For example, Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application. Figure 1 In the corresponding embodiments, mobile phones are used as examples of electronic devices and hair strands are used as examples of edge features. These examples do not constitute a limitation on the embodiments of this application.

[0077] like Figure 1 The scene shown in a can include a person in the foreground and a sun and sunlight in the background. At least one hair area can be displayed around the person's head, such as hair area 101 and hair area 102.

[0078] In response to a user opening the portrait mode in their camera, the electronic device can display something like... Figure 1 The interface shown as b in the diagram can be the preview interface for the portrait photography function. This interface may include one or more of the following: a shutter button 103, a preview screen, controls for switching cameras, or controls for indicating the use of the photography function. The camera application may also include other functions besides portrait photography, such as aperture shooting, night scene shooting, video recording, short videos, and HDR photography.

[0079] The preview image 104 displayed in the preview screen can be obtained by the electronic device recognizing the foreground and background of the captured image and blurring the recognized background. In a possible implementation, the blurred preview image may not be displayed in the preview screen, and this embodiment of the application does not limit this.

[0080] In response to the user's triggering operation on the camera button 103, the electronic device... Figure 1 Take a picture of the scene shown in Figure 'a' and obtain images such as... Figure 1 Image 105 is shown in the interface shown in 'c'. Image 105 can be obtained by the electronic device recognizing the foreground and background of the captured image and blurring the recognized background.

[0081] The captured image 105 may include: a clear hair area 101', a blurred hair area 102', and a clear partial background area 106, etc. (See also...) Figure 1 The scene in 'a' and Figure 1 In the photographed image shown in c, the hair region 101 can be part of the foreground region, so the hair region 101' can be a clear region; the hair region 102 can be part of the foreground, but due to the electronic device's misidentification of the foreground, the hair region 102 is mistakenly identified as the background, so the hair region 102' is blurred; the sun and its rays can be the background, but due to the electronic device's misidentification of the background, some sunlight rays are mistakenly identified as the foreground, so some background regions 106 are not blurred.

[0082] Understandably, due to the limited ability of electronic devices to cut out edge features, the blurred image is very prone to false blurring of edges or false sharpness of the background, affecting the clarity of the photographed image.

[0083] In view of this, embodiments of this application provide an image processing method that enables an electronic device to acquire a mask image and a ternary image corresponding to a clear image. The mask image is used to determine the position of edge features, and the ternary image is used to determine a transition region including the edge features. The electronic device can then extract the edge region corresponding to the position of the edge features from the clear image based on the position of the edge features, and fuse the edge region into a blurred image. The edge region is located within the transition region, and the blurred image can be obtained by blurring the background of the clear image.

[0084] Therefore, in order to better understand the embodiments of this application, the structure of the electronic device of the embodiments of this application will be described below. For example, Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0085] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, an indicator 192, a camera 193, and a display screen 194, etc.

[0086] The sensor module 180 may include one or more of the following: a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, or a bone conduction sensor. Figure 2 (not shown in the text), etc., are not specifically limited in this embodiment.

[0087] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0088] The processor 110 may include one or more processing units. These processing units may be independent devices or integrated within one or more processors. The processor 110 may also include a memory for storing instructions and data. For example, the processor 110 is used to implement the steps performed in the image processing method provided in the embodiments of this application, and to store instructions and data related to the image processing method.

[0089] The charging management module 140 receives charging input from the charger. The charger can be a wireless charger or a wired charger. The power management module 141 connects the charging management module 140 to the processor 110.

[0090] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0091] Electronic devices utilize GPUs, displays (194), and application processors to achieve display functions. The GPU is a microprocessor for image processing, connecting the displays (194) and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. For example, the GPU is used to execute the graphics rendering process within an image processing method.

[0092] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. In some embodiments, an electronic device may include one or N displays screens 194, where N is a positive integer greater than 1. For example, display screen 194 is used to display preview images and captured images in a camera application.

[0093] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0094] Camera 193 is used to capture still images or videos. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1. For example, in response to a user's photo-taking operation, camera 193 may be used to acquire a sequence of raw images.

[0095] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The internal memory 121 can be used to store executable program code, including instructions. The internal memory 121 may include a program storage area and a data storage area. For example, the internal memory 121 can be used to store executable program code for an image processing method.

[0096] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0097] A touch sensor can be located on the display screen 194, and the touch sensor and the display screen 194 together form a touch screen, or "touchscreen". For example, the touch sensor is used to receive the user's trigger operation on the camera button.

[0098] Button 190 includes a power button, volume buttons, etc. Button 190 can be a mechanical button or a touch button. The electronic device can receive button input and generate key signal inputs related to user settings and function control of the electronic device. In some scenarios, the electronic device can also respond to the user's action on one or more buttons of button 190 to perform a photo-taking operation. The specific method of photo-taking operation is not limited in this embodiment.

[0099] The software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc., which will not be elaborated here.

[0100] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other. The same or similar concepts or processes may not be described again in some embodiments.

[0101] Figure 3 This is a schematic diagram of the software structure of the electronic device provided in the embodiments of this application.

[0102] A layered architecture divides the system into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: application (APP), application framework layer (FWK), hardware abstraction layer (HAL), driver layer, and hardware layer.

[0103] The application layer may include a series of application packages. In this embodiment, the application package may include: a camera, a photo library, etc. The camera can capture images and display them. The photo library, also known as a photo album, can store and access captured images.

[0104] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment, the application framework layer may include a camera access interface, which may include camera management and camera devices. The camera access interface is used to provide application programming interfaces and programming frameworks for camera applications.

[0105] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0106] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. The camera algorithm library may include runtime code and data that implement the image processing methods provided in the embodiments of this application.

[0107] The driver layer is the layer between hardware and software. It includes drivers for various hardware components, such as camera drivers, digital signal processor drivers, and image processor drivers.

[0108] The camera device driver is used to drive the camera sensor to acquire images and to drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0109] The image processing method in this application embodiment will be described in detail below with reference to the above system structure:

[0110] In response to a user's action of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface in the application framework layer to launch the camera application. It then sends a command to start the camera by calling the camera device (Camera Device 1 and / or other camera devices) in the camera hardware abstraction layer. The camera hardware abstraction layer sends this command to the camera device driver in the kernel layer. This camera device driver can then start the corresponding camera sensor and acquire image light signals through the sensor. One camera device in the camera hardware abstraction layer corresponds to one camera sensor in the hardware layer.

[0111] Then, the camera sensor can transmit the acquired image light signal to the image signal processor for preprocessing to obtain the image electrical signal (raw image), and transmit the raw image to the camera hardware abstraction layer through the camera device driver.

[0112] The camera hardware abstraction layer can send the raw image to the camera algorithm library. The camera algorithm library stores program code that implements the image processing methods provided in the embodiments of this application. Based on a digital signal processor and an image processor, the camera algorithm library executes the above code to implement the process of generating the target image in the image processing methods described in the embodiments of this application.

[0113] The camera algorithm library can detect and send the raw images captured by the camera to the camera hardware abstraction layer. The camera hardware abstraction layer can then display these images.

[0114] It is understood that the software architecture provided in this application is only an example and does not constitute a limitation on the embodiments of this application.

[0115] For example, Figure 4 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 4 In a corresponding embodiment, the electronic device may include: a camera, a gallery, a camera access interface, a camera algorithm library, a camera hardware abstraction layer, and a camera device driver. The function of any of these modules can be found in [reference needed]. Figure 3 The corresponding implementation examples will not be described in detail here.

[0116] exist Figure 4In the corresponding embodiment, the edge feature is taken as the hair strand in the portrait as an example for illustration. In possible implementations, the edge feature may also include: the yarn in the sweater, or the hair of an animal, or the branches and leaves of a plant, etc., which are not limited in this embodiment.

[0117] like Figure 4 As shown, the image processing method may include the following steps:

[0118] S401, In response to the photo-taking operation, the camera device driver acquires the raw image.

[0119] The original image may include one or more of the following: long frames, short frames, or normal frames, wherein the exposure time of any frame in a long frame is greater than the exposure time of any frame in a normal frame, and the exposure time of any frame in a normal frame is greater than the exposure time of any frame in a short frame.

[0120] It is understood that the original image may include at least two image frames captured at the same time with different exposure times, so that the electronic device can generate a photographic image with HDR effect based on the image fusion of at least two image frames with different exposure times in an HDR photography scene.

[0121] The response to the photo-taking operation includes: responding to the user's trigger operation on the photo-taking button in the portrait photo-taking function, or responding to the user's trigger operation on the photo-taking button in the HDR photo-taking function, or responding to the user's trigger operation on the photo-taking button in the large aperture photo-taking function, or responding to the user's trigger operation on the record button in the video recording function, etc. The applicable scenarios for the image processing method are not limited in the embodiments of this application.

[0122] S402, the camera algorithm library obtains the raw image from the camera device driver.

[0123] Specifically, the camera hardware abstraction layer can obtain raw images from the camera device driver, and the camera algorithm library can obtain raw images from the camera hardware abstraction layer.

[0124] S403, the camera algorithm library, processes the raw image into a sharp image.

[0125] The process by which a camera algorithm library processes a raw image into a sharp image can be as follows: The camera algorithm library can acquire at least two images with different exposure times at the same moment from the raw image, and perform image preprocessing on any frame of these at least two images. Then, the camera algorithm library can perform image fusion on at least two frames acquired at the same moment from the at least two preprocessed images to obtain a sharp image. The sharp image can also be referred to as the first image.

[0126] It is understandable that since the sharp image is obtained by fusing at least two images with different exposure times, the sharp image has an HDR effect.

[0127] Image preprocessing may include one or more of the following: bad pixel correction processing, RAW domain noise reduction processing, black level correction processing, optical shadow correction processing / automatic white balance processing, color interpolation processing, color correction processing, or gamma correction processing, etc., which are not limited in this embodiment.

[0128] S404, the camera algorithm library, performs depth calculations on the sharp image to obtain the blurred image corresponding to the sharp image.

[0129] A blurred image can include a sharp foreground and a blurred background. When a sharp image is displayed as... Figure 1 As shown in 'a', after blurring the sharp image, a blurred image with a sharp foreground and a blurred background can be obtained. The blurred image can be... Figure 1 The image displayed in the interface shown in 'c'.

[0130] Camera algorithm libraries can determine the depth range based on the focus point and the depth image to obtain a depth image. For example, the depth at the focus point can be a first depth, and the depth of field at that focus point in the depth image can be 0. The depth of field at other locations in the depth image can be the difference between the depth at those other locations and the first depth, thus obtaining the foreground and background. The foreground can include the area in sharpness before the focus point, and the background can include the area in sharpness after the focus point.

[0131] The focus point can be determined by the camera's algorithm library based on a person or object captured in a clear image. For example, in a camera's portrait function, the electronic device can detect the subject being photographed. If a person is detected among the subjects, the focus point is set at the location of the person, thus focusing the image and making the person clearly visible in the frame.

[0132] A depth image may include depth information of any pixel. The depth information can represent the distance from each point in the scene to the camera plane and can reflect the geometry of visible surfaces in the scene. The depth information of any pixel in the depth image can be determined based on monocular depth estimation, binocular depth estimation, or depth estimation based on deep learning. This application does not limit this.

[0133] The methods for blurring can include Gaussian blurring or neural network-based blurring, etc., and are not limited in this embodiment. For example, the construction of a blurred image based on Gaussian blurring is illustrated as follows: The camera algorithm library can perform Gaussian blurring with random intensity on the entire image to obtain a Gaussian blur map. Then, by using the position of the foreground of the portrait in the mask image corresponding to the clear image, the clear foreground of the portrait is extracted from the clear image and replaced in the Gaussian blur map to obtain the blurred image.

[0134] S405, The camera algorithm library obtains the first mask image corresponding to the clear image and the first ternary image corresponding to the clear image.

[0135] There is a correspondence between the pixels in the first mask image and the pixels in the clear image, and there is also a correspondence between the pixels in the first ternary image and the pixels in the clear image.

[0136] For example, the camera algorithm library can pre-configure a target neural network (or a first model) to output a mask image and a ternary image corresponding to the image. For instance, the camera algorithm library can input a sharp image into the target neural network, which can output a first mask image and a first ternary image corresponding to the sharp image. The training process of the target neural network can be found in [reference needed]. Figure 5 Corresponding implementation examples.

[0137] The target neural network can include one or more of the following branches: a shared editor, a semantic segmentation branch, a detail prediction branch, or a fusion branch. Specifically, the camera algorithm library can input a sharp image into the target neural network. The shared editor can acquire the image features corresponding to the sharp image and input these features into the semantic segmentation branch to obtain the foreground of the human figure. The camera algorithm library acquires the Laplacian high-frequency map corresponding to the sharp image and inputs it into the detail prediction branch to obtain hair details within the transition region. The camera algorithm library inputs the foreground of the human figure and the hair details within the transition region into the fusion branch to obtain a first mask map and a first ternary map corresponding to the sharp image. The first mask map can include a foreground of the human figure with clear edge features, and the first ternary map can include a transition region, which may include hair areas.

[0138] Figure 5 This is a schematic diagram of the architecture of a target neural network provided in an embodiment of this application. Figure 5 As shown, the target neural network can be a type of neural network based on supervised learning.

[0139] Shared encoders are used to extract image features, such as acquiring image features from training images, which can be high-resolution images. For example, a shared encoder can use a pre-trained HRNet48 model on ImageNet.

[0140] The semantic segmentation branch is used to obtain the approximate foreground of the human figure in the training image. For example, the semantic segmentation branch can consist of five blocks, each block consisting of three ConvBNReLU modules and a bilinear upsampling module. The three ConvBNReLU modules can include: Conv, Batch Normalization (BN), and ReLU.

[0141] The detail prediction branch is used to predict more high-frequency information of edges in the training image, such as obtaining high-precision hair details in the training image. For example, the detail prediction branch can be composed of three residual blocks and convolutional layers.

[0142] Specifically, the training device can use the Laplacian edge detection method to extract details from edge regions in the training image, obtaining a Laplacian high-frequency map. The image features output by the shared encoder and the Laplacian high-frequency map are then fed into the detail prediction branch, which can predict a lot of high-frequency information about edges, such as predicting hair details in the training image.

[0143] The fusion branch is used to fuse the foreground of the human figure output by the semantic segmentation branch and the hair details output by the detail prediction branch, and output the predicted mask map and the predicted ternary map.

[0144] For example, the training data for the target neural network may include: training images, ground truth mask values, and ground truth ternary images. After the training device inputs the training images into the first neural network, the first neural network can adjust its parameters based on the differences between the predicted mask and the ground truth mask values, and the differences between the predicted ternary images and the ground truth ternary images. The first neural network can also be referred to as the source model.

[0145] For example, a well-trained fusion branch or a well-trained target neuron is obtained when the difference between the predicted mask and the ground truth mask satisfies the loss function loss1, and the difference between the predicted ternary image and the ground truth ternary image satisfies loss2. Here, loss1 can be the absolute value loss between the ground truth mask and the predicted mask; loss2 can be the cross-entropy loss between the ground truth ternary image and the predicted ternary image.

[0146] Both the predicted mask and the preset ternary map can be determined by the three branches: speech segmentation, detail prediction, and fusion.

[0147] In a possible implementation, since the semantic segmentation branch performs relatively coarse semantic segmentation of the image, only obtaining a rough foreground of the person, while the edges of the ground truth mask are more accurate, the training device can calculate loss4 on both the ground truth mask and the output of the semantic segmentation branch to ensure that the semantic segmentation branch outputs an image consistent with the foreground contours in the ground truth mask. For example, the training device can perform Gaussian blur processing on the ground truth mask to obtain a Gaussian blurred image of the ground truth mask edges (or simply a blurred mask edge image). By analyzing the difference between the blurred mask edge image and the output of the semantic segmentation branch, the parameters in the semantic segmentation branch are adjusted, and the training of the semantic segmentation branch is completed when the blurred mask edge image and the output of the semantic segmentation branch satisfy loss4. Here, loss4 can be considered as the structural similarity loss between the blurred mask edge image and the output of the semantic segmentation branch.

[0148] In a possible implementation, to enable the detail prediction branch to predict more details within the transition region containing edge features, the training device can calculate loss3 on the ground truth of the ternary image and the output of the detail prediction branch. For example, the training device can obtain a mask image (or simply transition region mask image, or third image) within the transition region of the ternary image based on the ground truth of the mask image and the ground truth of the ternary image. By analyzing the difference between the transition region mask image and the output of the detail prediction branch, the parameters in the detail prediction branch are adjusted. Training of the detail prediction branch is completed when the output of the transition region mask image and the output of the detail prediction branch satisfy loss3. Here, loss3 can be the L1 loss that limits the output of the transition region mask image and the detail prediction branch. L1 loss is also known as mean absolute error.

[0149] Understandably, electronic devices can train a better semantic segmentation branch based on loss4, and / or a better detail prediction branch based on loss4.

[0150] The fusion branch combines the foreground contour from the blurred mask edge map output by the semantic segmentation branch with the sharp edge features from the sharp feature map output by the detail prediction branch to obtain a predicted mask map, which then possesses sharp edge features. Furthermore, the fusion branch combines the foreground contour from the blurred mask edge map output by the semantic segmentation branch with the transition regions containing edge features from the sharp feature map output by the detail prediction branch to obtain a predicted ternary map. This predicted ternary map can be marked with transition regions, which may contain edge features.

[0151] Figure 5The training device described herein may be the electronic device described in the embodiments of this application, or it may be other devices or servers that can establish a connection with the electronic device, etc., and there is no limitation on this in the embodiments of this application.

[0152] Combination Figure 6 The corresponding embodiments illustrate the mask image and the ternary image. Figure 6 This is a schematic image provided for an embodiment of this application. For example, a clear image can be found in [the image provided]. Figure 6 The image shown in 'a', and the mask image can be found in [reference needed]. Figure 6 The image shown in b is a ternary plot; see [reference needed]. Figure 6 The image shown in c is shown in the image above.

[0153] S406, the camera algorithm library uses the clear image, the first mask image, and the first ternary image to optimize the hair region in the blurred image to obtain the target image.

[0154] Understandably, the camera algorithm library can extract a first region composed of sharp edge features from the sharp image based on the position of edge features in the first mask image, and then fuse the first region into the blurred image (or, in other words, overlay the first region onto the blurred image) to obtain the target image. It is necessary to ensure that the edge features in the first mask image can be located within the transition region of the first ternary image.

[0155] For example, camera algorithm libraries can also acquire target images based on generative networks. The generative network can be a generative adversarial network (GAN), which includes at least one generator for generating hair strands and at least one discriminator for determining whether the hair strands are real or fake. This application embodiment does not specifically limit the implementation details of the generative network.

[0156] The generative network can be trained based on: a training image, a predicted mask, a predicted ternary image, and a corresponding blurred image of the training image. The trained generative network can output a blurred image with clear edge features based on the training image, the predicted mask, the predicted ternary image, and the corresponding blurred image. The predicted ternary image can be used to control the range of edge features during the training process of the generative network.

[0157] For example, the camera algorithm library can input a sharp image, a first mask image, and a first ternary image into a generator network to generate an intermediate image. Further, the camera algorithm library can replace the region below the transition area indicated by the first ternary image in the intermediate image into the blurred image to obtain the target image. Since the final output image only replaces the hair transition region, the computational scope of the hair optimization module is smaller, and the computation speed is faster.

[0158] For example, a camera algorithm library can input a clear image, a first mask image, a first ternary image, and a blurred image into a generative adversarial network to output a target image, which may include clear edge features, such as clear hair strands.

[0159] After the camera algorithm library determines the target image, it can store the target image in the image library through the steps shown in S407-S408, and process the target image into a thumbnail and display the thumbnail through S409-S411. The embodiments of this application do not limit the order of the above two processes.

[0160] S407: The image library obtains the target image from the camera algorithm library.

[0161] Specifically, the camera access interface (or the first interface) can obtain a thumbnail from the camera algorithm library, and the camera can then obtain that thumbnail from the camera access interface (or the first interface). The first interface (in...) Figure 4 (Not shown in the image) can be used to establish a data path between the image library and the camera algorithm library. The type of the first interface is not limited in the embodiments of this application.

[0162] S408, The image library stores the target image.

[0163] Following S408, in response to a user opening a gallery, the electronic device can display the target image.

[0164] S409, the camera algorithm library processes the target image into a thumbnail.

[0165] Camera algorithm libraries can process target images into thumbnails using thumbnail processing methods such as sampling or neural networks, and this application does not limit this.

[0166] S410: The camera retrieves thumbnails from the camera algorithm library.

[0167] Specifically, the camera access interface can obtain thumbnails from the camera algorithm library, and then the camera can obtain the thumbnails from the camera access interface.

[0168] S411, The camera calls the display screen to show thumbnails.

[0169] For example, the camera can Figure 1 The thumbnail is displayed in the lower left corner of the interface shown by b.

[0170] Based on this, the electronic device can acquire a mask image and a ternary image corresponding to the sharp image. The mask image is used to determine the location of edge features, and the ternary image is used to determine the transition region including the edge features. Then, the electronic device can extract the edge region corresponding to the location of the edge features from the sharp image based on the edge feature location, and fuse the edge region into the blurred image. The edge region is located within this transition region, and the blurred image can be obtained by blurring the background of the sharp image.

[0171] Understandable Figure 4 The sequential relationship between the steps described herein is merely an example and should not be construed as limiting the embodiments of this application.

[0172] exist Figure 4 Based on the corresponding embodiments, this application provides a method for generating a ternary graph, such that the transition region in the ternary graph can include edge features.

[0173] For example, Figure 7 This is a schematic diagram illustrating the steps for generating a ternary graph according to an embodiment of this application. Figure 7 In the corresponding embodiments, the edge feature is taken as a hair strand as an example for illustration. This example does not constitute a limitation on the embodiments of this application.

[0174] Understandable Figure 7 The method for generating ternary graphs described herein can be used to generate ground truth values ​​for ternary graphs in a first neural network, for example, based on training images and mask image ground truth values. The process for generating ternary graph ground truth values ​​can be found below. Figure 7 The description in the text.

[0175] like Figure 7 As shown, the method for generating a ternary graph may include the following steps:

[0176] S701, the electronic device obtains the first hair region in the training image based on the semantic segmentation method.

[0177] For example, an electronic device can perform calculations on a training image using the Parsing-Net deep learning network to obtain the approximate hair distribution area, thus obtaining the first hair region. The first hair region can also be called the second region.

[0178] Parsing-Net is a multi-class deep learning method based on semantic segmentation, which can classify and label different regions of the human body, such as hair, face, hands, torso, and legs. The labeling results obtained by such methods are usually low in accuracy. This application does not specifically limit the method for obtaining the approximate distribution area of ​​hair strands.

[0179] S702. The electronic device performs connected component filtering to remove noise in the first hair region to obtain the second hair region.

[0180] Connected component filtering refers to finding pixels in an image that have the same or similar color values ​​and grouping them into a continuous region.

[0181] For example, an electronic device can obtain at least one connected component from the first hair region, and remove connected components with areas smaller than a first value from the at least one connected component to obtain the second hair region. The second hair region can also be called the third region. The first value can be a value such as 1 / 2500 of the area of ​​the training image.

[0182] Understandably, since the output of the Parsing-Net method has low accuracy and contains noise, connected component filtering can be used to remove noisy regions with an area smaller than 1 / 2500 of the training image area to improve the accuracy of hair region recognition.

[0183] S703, the electronic device expands the second hair region to obtain the third hair region.

[0184] The electronic device can perform 12 circular expansions of the second hair region, each with a kernel size of 6×6, to obtain the third hair region. The third hair region can also be called the fourth region.

[0185] Understandably, to improve the accuracy of hair region recognition and avoid unmarked hair regions in the output of the Parsing-Net method, electronic devices can perform dilation processing so that the dilated hair region can cover all hair details. However, in this embodiment, the number of dilation cycles, the shape of the dilation, and the size of the dilation are not limited.

[0186] S704. The electronic device expands the transition region in the true value of the mask to obtain a second mask.

[0187] The true value of the mask image can be determined based on the target neural network, or it can be determined by one or more methods such as manual annotation, data synthesis, or computer graphics (CG) synthesis.

[0188] The electronic device can perform a single circular expansion of the transition region in the ground truth mask image with a kernel size of 30×30 to obtain a second mask region. This second mask region can also be referred to as the expanded mask image. In this embodiment, the number of expansions, the shape of the expansion, and the size of the expansion are not limited.

[0189] S705, the electronic device takes the intersection of the second mask region and the third hair region to obtain the transition region in the ternary diagram.

[0190] Understandably, since the transition region of the mask image can encompass the entire body, it is necessary to find the intersection of the second mask image and the third hair region, and then perform another connected component filtering to remove noise regions with an area smaller than 1 / 20 of the entire image area. This removes the transition region that is not distributed in the hair region, resulting in the transition region in the ternary image. The transition region in the ternary image can also be called the first transition region.

[0191] S706. The electronic device sets the alpha=255 region from the mask image ground truth to the foreground of the ternary image, thus obtaining the ternary image ground truth. The electronic device can fuse the alpha=255 region from the mask image ground truth and the third hair region to obtain the ternary image ground truth.

[0192] Based on this, electronic devices can obtain the true value of a ternary map that restricts the transition region to the hair position by fusing the mask region and the hair region.

[0193] In possible implementations, the embodiments of this application may also be based on Figure 7 The ternary graph generation method described in [the text] obtains the first ternary graph corresponding to a clear image. The specific process is as follows: Figure 7 Similar to the case in China, I will not repeat it here.

[0194] exist Figure 4 Based on the corresponding embodiments, in order to clearly illustrate the embodiments of this application, an image processing method is also provided. Figure 8 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 8 In the corresponding embodiments, the edge feature is taken as a hair strand as an example for illustration. This example does not constitute a limitation on the embodiments of this application.

[0195] The camera algorithm library of an electronic device may include: a depth calculation module, a hair matting module, and a hair optimization module. The depth calculation module is used to obtain a blurred image with a clear foreground and a blurred background. The hair matting module is used to obtain a first mask image containing the hair strands and a first ternary image containing the hair strands. The hair optimization module is used to obtain a target image with clear hair strands using the clear image, the blurred image, the mask image, and the ternary image.

[0196] The implementation details of the depth calculation module can be found in S404, the implementation details of the hair cutout module can be found in S405, and the implementation details of the hair optimization module can be found in S406. They will not be repeated here.

[0197] The above combination Figures 4-8The methods provided in the embodiments of this application have been described. The apparatus for executing the above methods, provided in the embodiments of this application, is described below. Figure 9 As shown, Figure 9 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. The image processing device may be an electronic device in the embodiment of this application, or it may be a chip or chip system within an electronic device.

[0198] like Figure 9 As shown, the image processing apparatus 900 can be used in communication equipment, circuits, hardware components, or chips. The image processing apparatus 900 includes a display unit 901 and a processing unit 902. The display unit 901 supports the display steps performed by the display method; the processing unit 902 supports the information processing steps performed by the image processing apparatus 900.

[0199] In one possible implementation, the image processing apparatus 900 may further include a communication unit 903, which supports the image processing apparatus 900 in performing steps such as receiving or sending messages.

[0200] The image processing apparatus described in the embodiments of this application may include all of the following: Figure 9 The units described in the corresponding embodiments.

[0201] Specifically, the processing unit 902 can be integrated with the display unit 901, and the processing unit 902 and the display unit 901 may communicate with each other.

[0202] In one possible implementation, the image processing apparatus 900 may further include a storage unit 904. The storage unit 904 may include one or more memories, which may be devices in one or more devices or circuits used to store programs or data.

[0203] The storage unit 904 can exist independently or be connected to the processing unit 902 via a communication bus. Alternatively, the storage unit 904 can be integrated with the processing unit 902.

[0204] Taking the image processing device 900 as an example, which may be a chip or chip system of the electronic device in the embodiments of this application, the storage unit 904 may store computer-executable instructions for the methods of the electronic device, so that the processing unit 902 executes the methods of the electronic device in the above embodiments. The storage unit 904 may be a register, cache, or random access memory (RAM), etc., and the storage unit 904 may be integrated with the processing unit 902. The storage unit 904 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 904 may be independent of the processing unit 902.

[0205] In one possible implementation, the image processing apparatus 900 may further include a communication unit 903. The communication unit 903 supports interaction between the image processing apparatus 900 and other devices. For example, when the image processing apparatus 900 is an electronic device, the communication unit 903 may be a communication interface or interface circuit. When the image processing apparatus 900 is a chip or chip system within an electronic device, the communication unit 903 may be a communication interface. For example, the communication interface may be an input / output interface, pins, or circuits.

[0206] The apparatus in this embodiment can be used to execute the steps performed in the above method embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0207] Figure 10 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application.

[0208] The electronic device includes a processor 1001, a communication line 1004, and at least one communication interface. Figure 10 (The example described uses communication interface 1003 as an example).

[0209] The processor 1001 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0210] The communication line 1004 may include circuitry for transmitting information between the aforementioned components.

[0211] Communication interface 1003 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0212] Possibly, the electronic device may also include a memory 1002.

[0213] The memory 1002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory may exist independently and be connected to the processor via communication line 1004. The memory may also be integrated with the processor.

[0214] The memory 1002 stores computer execution instructions for implementing the scheme of this application, and the processor 1001 controls the execution. The processor 1001 executes the computer execution instructions stored in the memory 1002 to implement the method provided in the embodiments of this application.

[0215] It is possible that the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0216] In a specific implementation, as one example, the processor 1001 may include one or more CPUs, for example... Figure 10 CPU0 and CPU1 in the CPU.

[0217] In a specific implementation, as one example, an electronic device may include multiple processors, for example... Figure 10Processors 1001 and 1005 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0218] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0219] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0220] This application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium capable of transferring a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0221] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may also include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers.

[0222] The above combinations should also be included within the scope of computer-readable media. The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0223] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. An image processing method, characterized in that, The method includes: In response to the photo-taking operation, the first image is captured; Obtain the blurred image corresponding to the first image; wherein, the blurred image is obtained by blurring the background in the first image; The first image is input into a first model, which outputs a mask and a ternary image corresponding to the first image. The first model includes a shared editor, a semantic segmentation branch, a detail prediction branch, and a fusion branch. The shared editor extracts image features from the first image. The semantic segmentation branch performs semantic segmentation on the first image and obtains the foreground. The detail prediction branch obtains edge features from the first image. The fusion branch fuses the foreground and edge features from the first image to obtain the mask and ternary image corresponding to the first image. The mask includes the positions of the edge features, and the ternary image includes transition regions obtained based on the edge features. Using the position of the edge features obtained based on the mask image, a first region composed of the edge features is determined from the first image, and the first region is overlaid on the blurred image to obtain a target image; wherein the first region is located within the transition region.

2. The method according to claim 1, characterized in that, The method further includes: The first model was trained in the following way: The training images are input into the shared editor to obtain the image features of the training images; The image features of the training image are input into the semantic segmentation branch to obtain the output result of the semantic segmentation branch; Obtain the Laplacian high-frequency map corresponding to the training image, input the Laplacian high-frequency map into the detail prediction branch, and obtain the output result of the detail prediction branch; The output of the semantic segmentation branch and the output of the detail prediction branch are input into the fusion branch to obtain the predicted mask map and the predicted ternary map. The first trained model is obtained when the difference between the predicted mask image and the ground truth mask image satisfies the first loss function, and the difference between the predicted ternary image and the ground truth ternary image satisfies the second loss function.

3. The method according to claim 2, characterized in that, The method further includes: Gaussian blurring is applied to the edges of the ground truth of the mask image to obtain the second image; The trained semantic segmentation branch is obtained when the difference between the second image and the output of the semantic segmentation branch satisfies the fourth loss function.

4. The method according to claim 2 or 3, characterized in that, The method further includes: Based on the transition region in the ternary image ground truth, the image within the transition region is obtained from the mask image ground truth to obtain the third image; The trained detail prediction branch is obtained when the difference between the third image and the output of the detail prediction branch satisfies the third loss function.

5. The method according to claim 2, characterized in that, The method further includes: A second region containing edge features is obtained from the training image based on a semantic segmentation method; The third region is obtained by filtering the connected components of the second region. The third region is expanded to obtain the fourth region; The ground truth of the mask image is dilated to obtain the dilated mask image; The first transition region is obtained by acquiring the intersection of the expanded mask image and the fourth region. The true value of the ternary image is determined based on the first transition region and the foreground in the expanded mask image.

6. The method according to claim 1, characterized in that, The edge features include one or more of the following: human hair strands, yarn in a sweater, animal hair, or plant branches and leaves.

7. The method according to claim 1, characterized in that, The process of capturing a first image in response to a photo-taking operation includes: In response to the photo-taking operation, at least two images with different exposure times are acquired, and the images with different exposure times are fused to obtain the first image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it causes the computer to perform the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, causes the computer to perform the method as described in any one of claims 1-7.

11. A chip, characterized in that, include: A processor for reading instructions stored in a memory, and when the processor executes the instructions, causing the chip to implement the method described in any one of claims 1-7.

Images