Image processing method and electronic device

Abstract

An image processing method and electronic device, which performs corrosion, expansion, smoothing and other processing on a mask image used to segment the foreground and background, and then segments the foreground and background using a mask image, thereby eliminating burrs and jagged edges on the foreground portion of the image after segmentation. In addition, a fusion weight can be determined based on the mask image after smoothing, and the foreground and background after segmentation can be fused, so that the transition between the fused foreground and background is natural.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an image processing method and electronic device. Background Art

[0002] With the development of terminal technology, the processing ability of terminal devices for images is becoming more and more powerful. Users can use terminal devices to process the captured images to obtain the desired effects. One of the processes in the image processing is the foreground and background segmentation of the image. The segmented foreground image and background image are processed separately and then fused.

[0003] In the prior art, the image foreground and background segmentation generally uses a deep learning segmentation model to obtain a mask for foreground and background segmentation, and then the image is segmented into a foreground image and a background image based on the mask. In order to reduce the running time of the deep learning segmentation model, the original image to be processed is generally downsampled to a low-resolution image, and then the mask is obtained based on the low-resolution image through the deep learning segmentation model. In this way, the edges of the obtained mask may have burrs, resulting in burrs on the edges of the foreground part of the image that is finally fused after segmentation processing, affecting the visual effect. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and an electronic device, which are used to provide a way to eliminate burrs existing on the edge of the foreground part of an image after segmentation processing.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which can be executed by an electronic device. The method includes: obtaining a first mask image for segmenting the foreground and background of an image to be processed, wherein the first mask image is a binary mask image; performing a first process on the first mask image to obtain a second mask image, wherein the first process includes corrosion and / or expansion; and segmenting the foreground and background of the image to be processed based on the second mask image to obtain a first foreground image and a first background image of the image to be processed. By performing corrosion and / or expansion processing on the mask image, burrs on the foreground edge of the mask image can be eliminated.

[0006] In one possible design, the first processing also includes smoothing processing; the first processing of the first mask image to obtain the second mask image includes: corroding and / or dilating the first mask image to obtain a third mask image; smoothing the third mask image, and binarizing the mask image obtained after the smoothing processing according to a first threshold to obtain the second mask image.

[0007] Through the above design, the third mask image obtained after the erosion and / or dilation process is smoothed, which can further eliminate the jagged edges of the foreground of the mask image.

[0008] In a possible design, the method further includes: performing personalized processing on the first foreground image and / or the first background image in accordance with user needs to obtain a second foreground image and a second background image, and performing fusion processing on the second foreground image and the second background image. By using the processed mask image to segment the foreground and the background and then performing personalized processing and fusion, burrs at the foreground and background edges of the fused image caused by burrs on the foreground edge of the mask image can be eliminated, thereby bringing better visual effects.

[0009] In one possible design, the second foreground image and the second background image are fused, including: determining a fusion weight based on a mask image obtained after smoothing the second mask image, and fusing the second foreground image and the second background image based on the fusion weight so that the transition between the foreground and the background of the fused image is natural.

[0010] In a possible design, determining a first fusion weight according to a mask image obtained by smoothing the second mask image, and fusing the second foreground image and the second background image according to the first fusion weight includes:

[0011] The second foreground image and the second background image are fused by the following formula to obtain a fused image;

[0012] fusion i =mask i *foreground i +(1-mask i )*background i ;

[0013] Among them, fusion i represents the pixel value of the i-th pixel of the fused image, mask i represents the pixel value of the i-th pixel point of the mask image obtained after the smoothing process, foreground i represents the pixel value of the i-th pixel of the second foreground image, background i Represents the pixel value of the i-th pixel of the second background image.

[0014] In order to make the foreground image and the background image transition naturally after fusion, the mask image for segmenting the foreground image and the background image is smoothed, and then the pixel value mask of the smoothed mask image is converted intoi As the weight value of the foreground image, (1-mask i ) is used as the weight value of the background image, so as to perform weighted fusion processing on the foreground image and the background image, so that the edge of the foreground image can smoothly transition to the background image.

[0015] In one possible design, the first mask image is corroded and / or expanded. In one way, the first mask image may be corroded and then expanded. In another way, the first mask image may be expanded and then corroded.

[0016] In a possible design, the smoothing process is any one of the following processing modes: mean filtering, median filtering, bilateral filtering or Gaussian filtering.

[0017] In the second aspect, an embodiment of the present application also provides an image processing device, including units / modules for executing the methods described in the first aspect or any design of the first aspect; these modules / units can be implemented by hardware, or corresponding software can be implemented by hardware.

[0018] In the third aspect, an embodiment of the present application also provides an electronic device, including a processor and a memory; wherein the processor is coupled to the memory; wherein the memory is used to store program instructions; and the processor is used to read the program instructions stored in the memory to implement the method of the first aspect and any possible design thereof.

[0019] In a fourth aspect, an embodiment of the present application also provides a computer storage medium, which stores program instructions. When the program instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible design thereof.

[0020] In a fifth aspect, an embodiment of the present application also provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method of the first aspect and any possible design thereof.

[0021] In a sixth aspect, an embodiment of the present application also provides a chip, which is coupled to a memory in an electronic device to execute the method of the first aspect and any possible design thereof.

[0022] In addition, the technical effects brought about by the second to sixth aspects can be found in the description of the first aspect above and will not be repeated here.

[0023] It should be noted that, in the embodiments of the present application, “coupling” refers to two components being directly or indirectly coupled to each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the structure of a mobile phone 100 provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a user graphical interface of a mobile phone 100 provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a user graphical interface of a mobile phone 100 provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a user graphical interface of a mobile phone 100 provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the image processing method provided in the embodiment of the present application;

[0029] Figure 6 A schematic diagram of a mask image with burrs on the foreground edge provided in an embodiment of the present application;

[0030] Fig. 7A A schematic diagram of a mask image for removing burrs by etching provided in an embodiment of the present application;

[0031] Figure 7B A schematic diagram of a mask image subjected to expansion processing provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of a mask image with jagged edges on the foreground provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of a mask image after smoothing provided in an embodiment of the present application;

[0034] Fig.10 A schematic diagram of a second mask image after smoothing provided in an embodiment of the application;

[0035] Fig.11A A schematic diagram of a background image and a foreground image to be fused provided in an embodiment of the present application;

[0036] Fig. 11B A schematic diagram of a fused image obtained by smoothing edge processing provided in an embodiment of the present application;

[0037] Fig.12 A schematic diagram of the structure of the device 1200 provided in an embodiment of the present application;

[0038] Fig.13 A schematic diagram of the structure of an electronic device 1300 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application can be applied to electronic devices that can realize image segmentation and fusion, such as terminal devices, cloud servers, etc. One or more applications can be installed in the terminal device, and the application is a computer program that can realize one or more specific functions. For example, camera application, album application, WeChat, Tencent chat software (QQ), WhatsApp Messenger, Line, photo sharing (Instagram), Kakao Talk, DingTalk, etc. The application mentioned below can be an application that comes with the terminal device when it leaves the factory, or it can be an application that the user downloads from the network side during the use of the terminal device. The image processing method provided in the embodiments of the present application can be applied in one or more applications. For example, the camera application can use the image processing method provided in the embodiments of the present application to process the image.

[0040] The image involved in the embodiment of the present application may be a picture or a frame of image in a video stream, which is not limited in the embodiment of the present application.

[0041] The pixel involved in the embodiment of the present application is the smallest imaging unit on an image. A pixel can correspond to a coordinate point on the image. A pixel can include a parameter (such as grayscale) or a set of multiple parameters (such as grayscale, brightness, color, etc.). If a pixel includes a parameter, then the pixel value is the value of the parameter. If a pixel is a set of multiple parameters, then the pixel value includes the value of each parameter in the set.

[0042] The "multiple" involved in the embodiments of the present application refers to greater than or equal to two.

[0043] It should be noted that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the previous and subsequent associated objects are in an "or" relationship. In the description of the embodiments of the present invention, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0044] The following introduces a terminal device, a graphical user interface (GUI) for such a terminal device, and an embodiment for using such a terminal device. In some embodiments of the present application, the terminal device may be a portable terminal, such as a mobile phone, a tablet computer, a wearable device with wireless communication function (such as a smart watch), etc. The portable terminal includes a device (such as a processor) that can capture images and process the captured images. Exemplary embodiments of the portable terminal include but are not limited to a device equipped with Or a portable terminal with other operating systems. The portable terminal may also be other portable terminals, as long as it can capture images and process the captured images. It should also be understood that in some other embodiments of the present application, the terminal device may not be a portable terminal, but a desktop computer that can capture images and process the captured images. In some embodiments of the present application, the terminal device may also be a camera, a video camera, a camcorder, and other devices.

[0045] In other embodiments of the present application, the terminal device may not need to have an image acquisition function, but may have a communication function, that is, the terminal device can receive images sent by other devices and then process the received images; wherein the other devices may be network-side devices, such as images downloaded by the terminal device from the network; or, the terminal device receives images sent by other devices through the WeChat application or other communication applications.

[0046] In other embodiments of the present application, the terminal device may not have the image processing function, but may have the communication function. For example, after the terminal device captures an image, the image may be sent to other devices such as a server, and the other device processes the image using the image processing method provided in the embodiment of the present application, and then sends the processed image to the terminal device.

[0047] Take the mobile phone as an example. Figure 1 A schematic structural diagram of the mobile phone 100 is shown.

[0048] The mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 151, a wireless communication module 152, a sensor module 180, a button 190, a display screen 194, a camera 193, etc. It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0049] Below Figure 1 The components of the illustrated mobile phone 100 are described.

[0050] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. Among them, the controller can be the nerve center and command center of the mobile phone 100. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0051] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0052] The processor 110 can run the software code of the image processing algorithm provided in the embodiment of the present application and execute the image processing process below. The specific image processing process will be introduced later.

[0053] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store software codes of operating systems and applications (such as camera applications, album applications, WeChat applications, etc.). The data storage area can store personal data created during the use of the mobile phone 100 (such as images before processing, images after processing, etc.).

[0054] The internal memory 121 may also store software codes of the image processing method provided in the embodiment of the present application. When the processor 110 runs the code, the following image processing flow is executed.

[0055] The internal memory 121 may also store other contents. For example, the internal memory 121 may store mask images of the foreground and background of the image to be processed, and the processor 110 may process the mask images.

[0056] The internal memory 121 can also store processed mask images, fused images, etc. For example, the fused image and the original image (i.e., the image before segmentation and fusion) can be stored correspondingly. The internal memory 121 can include a high-speed random access memory and a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0057] The external memory interface 120 is used to connect the external memory to the mobile phone 100, wherein the external memory includes an external memory card (SD memory card) and a NAS storage device, etc., which is not limited in the embodiment of the present application. In order to save the storage space of the internal memory 121, the mobile phone 100 can also store the software code of the image processing method provided in the embodiment of the application, the processed image, etc. in the external memory. The processor 110 can access the data stored in the external memory through the external memory interface 120.

[0058] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc. Figure 1 Not shown.

[0059] Among them, the distance sensor is used to measure the distance. The mobile phone 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the mobile phone 100 can use the distance sensor to measure the distance to achieve fast focus. In other embodiments, the mobile phone 100 can also use the distance sensor to detect whether a person or object is approaching. The proximity light sensor may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The mobile phone 100 emits infrared light outward through the light emitting diode. The mobile phone 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the mobile phone 100. When insufficient reflected light is detected, the mobile phone 100 can determine that there is no object near the mobile phone 100. The mobile phone 100 can use the proximity light sensor to detect that the user holds the mobile phone 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor can also be used in the leather case mode and the pocket mode to automatically unlock and lock the screen.

[0060] The ambient light sensor is used to sense the brightness of the ambient light. The mobile phone 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor can also cooperate with the proximity light sensor to detect whether the mobile phone 100 is in a pocket to prevent accidental touches. The fingerprint sensor is used to collect fingerprints. The mobile phone 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc. The temperature sensor is used to detect temperature. In some embodiments, the mobile phone 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0061] The touch sensor is also called a "touch panel". The touch sensor can be arranged on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor can also be arranged on the surface of the mobile phone 100, which is different from the position of the display screen 194.

[0062] In addition, the mobile phone 100 can implement audio functions through an audio module, a speaker, a receiver, a microphone, a headphone interface, and an application processor. For example, music playback, recording, etc. The mobile phone 100 can receive input from the key 190 and generate key signal input related to the user settings and function control of the mobile phone 100. The mobile phone 100 can use a motor to generate a vibration prompt (such as a vibration prompt for an incoming call). The indicator in the mobile phone 100 can be an indicator light, which can be used to indicate the charging status, power changes, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface in the mobile phone 100 is used to connect a SIM card. The SIM card can be inserted into the SIM card interface or pulled out from the SIM card interface to achieve contact and separation with the mobile phone 100.

[0063] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0064] The embodiments of the present application can be applied to image processing scenarios including foreground and background, specifically, image processing in image capture, image processing in video recording, or application scenarios such as photo editing. For example, the foreground is a person, an animal, etc., and the background is a scene. It is necessary to perform different processing on the foreground and the background, such as performing personalized processing on the foreground and / or the background in accordance with user needs, for example, adjusting the color temperature and tone of the foreground and the background, or blurring the background, or processing light, adjusting the light effects of the foreground and the background, etc. The following is explained by taking the application to camera photography, image processing in the gallery, and photo editing software scenarios as examples.

[0065] In one example, the image processing provided by the embodiments of the present application can be applied to a camera photo-taking scenario.

[0066] Take adjusting the color tone of the background image as an example. The display screen 194 of the mobile phone 100 displays the main interface, where the main interface includes a camera icon. For example, the main interface can be Figure 2 The user interface 200 shown in (a) of FIG. 2 includes a camera icon 201. In addition, the user interface 200 may also include icons of other applications, such as a setting icon, a memo icon, a gallery icon, and the like.

[0067] The mobile phone 100 can respond to the user triggering the first operation of starting the camera icon 201, start the camera 193, and display the viewfinder interface 202 on the display screen 194, and the image captured by the camera 193 is displayed on the viewfinder interface 202. It is understandable that the camera 193 in the mobile phone 100 can also be turned on all the time. It should be understood that the application corresponding to the camera icon 201 can be called a camera, or it can be called other names, without limitation. The following embodiments refer to the application corresponding to the camera icon 201 as a camera. It should be noted that the first operation can be a click operation on the camera icon 201, or it can be an operation of receiving a voice command (such as "open the camera") issued by the user, or it can be a quick gesture operation (such as three-finger swipe, etc.). In addition, in the embodiment of the present application, when the mobile phone 100 is in a black screen or lock screen state, it can also respond to the user's voice command or quick gesture operation to directly start the camera 193 and display the viewfinder interface 202 on the display screen 194.

[0068] For example, Figure 2 Taking the viewfinder interface 202 shown in (b) of FIG. 1 as an example, the viewfinder interface 202 displays the image captured by the front camera of the mobile phone 100. The viewfinder interface 202 may also display the image captured by the rear camera of the mobile phone 100. It should be noted that the mobile phone 100 of the present application may respond to the Figure 2 The camera switch button 204 shown in (c) is operated to switch between the front camera and the rear camera.

[0069] The following is a detailed description of the case where the mobile phone 100 displays the viewfinder interface 202 on the display screen 194 in response to the first operation, and the image displayed in the viewfinder interface 202 is the image captured by the front camera of the mobile phone 100. Figure 2 As shown in (b) of FIG. 1 , the viewfinder interface may further include a shooting mode selection button area 203, which may include mode selection buttons such as photo, video, professional, portrait, large aperture, night scene, or more. The user may view more shooting modes of the mobile phone 100 by sliding left and right on the shooting mode selection button area 203. For example, if the "portrait" mode is selected, Figure 2 As shown in (c) of FIG. 1 , the viewfinder interface 202 may further include a color tone selection button 206. In response to the operation triggered by the user clicking the color tone selection button 206, the mobile phone 100 may prompt a plurality of color tones for the user to select, such as Figure 2 As shown in (c) in FIG. 8 , hue A and hue B are popped up for user selection as an example.

[0070] If the user clicks to select hue A, the mobile phone 100 segments the foreground and background of the image to be processed in the viewfinder interface 202 captured by the camera to obtain a first mask image, and then performs a first process on the first mask image to obtain a second mask image. For example, the first process may include corrosion and / or expansion; thereafter, the mobile phone 100 segments the foreground and background of the image to be processed based on the second mask image to obtain a first foreground image and a first background image of the image to be processed, and then performs personalized processing on the first foreground image and / or the first background image in accordance with user needs to obtain a second foreground image and a second background image, and finally fuses the second foreground image and the second background image to obtain a fused image. Finally, the image obtained by the fusion after processing is displayed in the viewfinder interface 202, and the user can see the image after the hue adjustment in the viewfinder interface 202. Here, the user can click the shooting button 205 after the image seen reaches the degree of satisfaction. In response to the user clicking the shooting button 205, the mobile phone 100 triggers the image captured and processed in the photo viewfinder interface 202, thereby obtaining an image with adjusted background hue.

[0071] As an example, if the user is not satisfied with the image after tone adjustment in the framing interface 202, he can continue to click the tone selection button 206, thereby triggering the mobile phone 100 to continue adjusting the framing image content according to the above process until the user is satisfied, and then click the shooting button 205.

[0072] In another example, the image processing provided by the embodiments of the present application can be applied to the scenario of processing images in a gallery.

[0073] See also Figure 3 As shown in (a) of FIG. 1 , the mobile phone 100 displays a main interface 301, which includes icons of multiple applications, including a gallery icon 302. When the mobile phone 100 detects a second operation on the icon 302, an interface 303 is displayed. Figure 3 As shown in (b) of FIG. 1 , the interface 303 includes thumbnails of multiple images. The mobile phone 100 detects an operation on the thumbnail 304 and displays the interface 305. Figure 3 As shown in (b) of the figure, the interface 305 includes an image 306 (the image corresponding to the thumbnail 304). When the mobile phone 100 detects the operation on the control 307, multiple options are displayed, such as the "tone adjustment mode" selection and the "filter" option. When the mobile phone 100 detects that the "tone adjustment mode" selection is selected and the tone A is selected, the processor 110 in the mobile phone 100 processes the image 306 using the image processing method provided in the embodiment of the present application. The specific image processing method can be found in the subsequent Figure 5 The description of the corresponding embodiment will not be repeated here. Specifically, the color tone of the background part of the image 306 is adjusted to the color tone A, and the image obtained by the processing and fusion is displayed on the display screen, such as Figure 3 As shown in (d) in .

[0074] In another example, the image processing provided by the embodiment of the present application can be applied to the photo editing software. It should be noted that the modification software can be installed when the mobile phone 100 leaves the factory, or the mobile phone 100 can download and install it from the network side. There can be many types of photo editing software, such as Meitu Xiuxiu, vsco, MIX, etc., or the photo editing function can also be integrated into the gallery in the above scenario 2, which is not limited by the embodiment of the present application.

[0075] For example, see Figure 4 As shown in (a), the mobile phone 100 displays an interface 401, and an image 402 is displayed in the interface 401. When the mobile phone 100 detects an operation on the editing control 403, two options are displayed: "tone adjustment mode" and filter mode. When the mobile phone 100 detects an operation on the "tone adjustment mode", the processor 40 in the mobile phone 100 processes the image 402 using the image processing method provided in the embodiment of the present application. The specific image processing method can be found in the subsequent Figure 5 The corresponding embodiment will not be described again here. The image obtained by processing and fusion is displayed on the display screen 194. Figure 4(b) in the figure. It should be understood that the embodiment of the present application only takes the adjustment of image tone as an example. Of course, it can also be background blur, light effect adjustment, etc. The scenes that require separate processing of foreground images and background images are all applicable to the present application. Three possible scenes are listed above. It should be noted that the method provided in the embodiment of the present application can also be used in other scenes, such as video recording scenes, WeChat video call scenes, WeChat emoticon package creation scenes, etc.

[0076] Of course, the above only lists two scenarios of shooting and processing gallery photos. The embodiments of the present application can also be used in the mobile phone 100 to receive images from other electronic devices, and perform the processing of the embodiments of the present application on the images, and then save the processed images locally or display them to the user, or feed the processed images back to the electronic device that sent the original image, which will not be described in detail here.

[0077] The following is a detailed description of how the device performs the image processing method provided in the embodiment of the present application. Figure 5 As shown, this process can be applied to Figure 1 The mobile phone 100 shown, or other devices. Figure 1 Taking the mobile phone 100 as an example, the software code of the method can be stored in the internal memory 121, and the processor 110 of the mobile phone 100 can run the software code to implement Figure 5 The image processing flow shown.

[0078] like Figure 5 As shown, the process of the method includes:

[0079] S501: The mobile phone 100 obtains a first mask image for segmenting the foreground and background of an image to be processed, where the first mask image is a binary mask image.

[0080] The image to be processed may be a picture captured by the mobile phone 100 through a camera, or a picture stored in the mobile phone 100 .

[0081] Exemplarily, the mobile phone 100 can obtain a first mask image of the image to be processed by a deep learning foreground-background segmentation algorithm. Among them, the area with a pixel value of 0 in the first mask image can represent the background area, and the area with a pixel value of 1 can represent the foreground area. Of course, the area with a pixel value of 1 can represent the background area, and the area with a pixel value of 0 can represent the foreground area. This application does not limit this. In the embodiment of the present application, it is taken as an example that the area with a pixel value of 0 in a mask image can represent the background area, and the area with a pixel value of 1 can represent the foreground area.

[0082] S502: The mobile phone 100 performs a first processing on the first mask image to obtain a second mask image. The first processing may include corrosion and / or expansion.

[0083] It should be noted that the corrosion process can eliminate the boundary points of the object, shrink the target, and eliminate the noise points smaller than the structural elements; the expansion process can merge all the background points in contact with the object into the object, enlarge the target, and fill the holes in the target.

[0084] Exemplarily, when the first processing includes corrosion and / or expansion, the mobile phone 100 may perform the corrosion and / or expansion processing on the first mask image in the following manner:

[0085] In one embodiment, the mobile phone 100 first performs an erosion process on the first mask image, and then performs an expansion process on the eroded first mask image.

[0086] In another manner, the mobile phone 100 first performs dilation processing on the first mask image, and then performs erosion processing on the first mask image after the dilation processing.

[0087] S503: The mobile phone 100 segments the foreground and background of the image to be processed based on the second mask image to obtain a first foreground image and a first background image of the image to be processed.

[0088] Since the deep learning segmentation model takes a long time to obtain the mask image, in order to improve the user experience, it is necessary to reduce the time it takes to obtain the mask image using deep learning. In addition, in the field of video processing, due to the real-time nature of video, the image processing of each frame must be completed within a limited time, otherwise it will cause frame loss or inconsistent effects between frames. Therefore, it is also necessary to reduce the time it takes to obtain the mask image using deep learning. Generally, the original image is downsampled to a low resolution, such as 288*288, and the output of the deep learning-based segmentation model is also low resolution, and then upsampled to restore the resolution of the original image. Perhaps because the original image has been downsampled and a lot of information has been lost, the deep learning-based segmentation model is likely to mistake part of the background as part of the foreground, causing the output foreground edge to contain burrs, see Figure 6 After the mask image obtained based on the segmentation model is corroded and expanded in the above manner, the burrs on the edge of the mask can be reduced. The original mask is first corroded to eliminate the burrs on the edge of the mask. For example, see Fig. 7A However, the mask is usually slightly smaller than the original mask. In this case, the eroded mask can be expanded, for example, see Figure 7BTherefore, after the original image is segmented based on the mask, the edges of the foreground image and the background image will not have burrs, achieving a natural visual effect.

[0089] The mask restored by upsampling may still have jagged edges due to upsampling. Figure 8 In order to eliminate the jagged edges of the first mask image obtained by the deep learning segmentation model, the first mask image can be smoothed (also called blurred), and then the mask image obtained after the smoothing process is binarized, and then the foreground and background of the image to be processed are segmented according to the mask image obtained by the binarization process.

[0090] In addition, after corrosion and / or expansion, although the burrs on the edge of the mask are eliminated, jagged edges may still exist. In this case, the first processing in the above step S502 may also include smoothing processing, and the third mask image obtained after corrosion and / or expansion processing may be smoothed (also called blurring processing), and then the mask image obtained after the smoothing processing is binarized by a set first threshold to obtain the second mask image. Furthermore, in step S503, the mobile phone 100 segments the foreground and background of the image to be processed according to the second mask image to obtain the first foreground image and the first background image of the image to be processed. There are no jagged edges on the mask image after smoothing and binarization, for example, see Fig. 9 shown.

[0091] Exemplarily, in the smoothing process of the mask image (such as the first mask image, the second mask image or the third mask image) involved in the embodiments of the present application, mean filtering, median filtering, Gaussian filtering, bilateral filtering, etc. can be used to process the image to achieve a smoothing effect.

[0092] After the mobile phone 100 segments the foreground and background of the image to be processed based on the second mask image to obtain the first foreground image and the first background image of the image to be processed, it executes S504, performs personalized processing on the first foreground image and / or the first background image in accordance with user needs to obtain the second foreground image and the second background image, and executes S505 to fuse the second foreground image and the second background image to obtain a fused image.

[0093] In addition, after obtaining the original image to be processed, the foreground and background are generally separated according to the mask and processed separately. When processing the background image, only the places where the mask is 0 are processed, and the pixel values ​​are directly set to 0 where the mask is not 0; when processing the foreground image, only the places where the mask is 1 are processed, and the pixel values ​​are directly set to 0 where the mask is not 1. After processing the foreground image and the background image, a processed foreground image and a processed background image are obtained respectively. Generally, the two images are directly added and fused to obtain the image processed image. When the processing results of the foreground image and the background image, such as hue, light, and clarity, are quite different, the simple addition operation results in a stiff transition, obvious regional edges, and a very unnatural visual effect.

[0094] In order to eliminate the problem of unnatural edge transition, the second mask image may be smoothed before the second foreground image and the second background image are fused. Fig.10 As shown, a fusion weight is determined based on the mask image obtained after smoothing the second mask image, and the second foreground image and the second background image are fused based on the fusion weight. The fusion weight here includes the weight value of the second foreground image and the weight value of the second background image.

[0095] Exemplarily, when the second foreground image and the second background image are fused according to the fusion weight, the second foreground image and the second background image may be fused to obtain a fused image by the following formula (1):

[0096] fusion i =mask i *foreground i +(1-mask i )*background i ;Formula (1)

[0097] Among them, fusion i represents the pixel value of the i-th pixel of the fused image, mask i represents the pixel value of the i-th pixel point of the second mask image after the smoothing process, foreground i represents the pixel value of the i-th pixel of the second foreground image, background i Represents the pixel value of the i-th pixel of the second background image.

[0098] From the above formula (1), it can be seen that in order to make the foreground image and the background image transition naturally after fusion, the mask image for segmenting the foreground image and the background image is smoothed, and then the pixel value mask of the smoothed mask image is converted into i As the weight value of the foreground image, (1-mask i ) as the weight value of the background image, so as to perform weighted fusion processing on the foreground image and the background image. This allows the edge of the foreground image to smoothly transition to the background image. For example, see Fig.11A The background image and foreground image to be fused are shown in FIG. 1 , and the fused image obtained by smoothing the edges of the embodiment of the present application is shown in FIG. Fig. 11B shown.

[0099] In the above embodiments provided by the present application, the method provided by the embodiment of the present application is introduced from the perspective of the terminal device (mobile phone 100) as the execution subject. In order to implement the various functions in the method provided by the above embodiments of the present application, the terminal may include a hardware structure and / or a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether a certain function of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0100] Based on the same concept as the above method, the present application embodiment also provides a device 12, see Fig.12 As shown, the device 1120 may be a processor in an electronic device, or a chip or a chip system, or a module in an electronic device. Schematically, the device may include an acquisition unit 1201, a mask processing unit 1202, a segmentation unit 1203, a personalized processing unit 1204, and a fusion unit 1205. The acquisition unit 1201, the mask processing unit 1202, the segmentation unit 1203, the personalized processing unit 1204, and the fusion unit 1205 perform Figure 5 The method steps shown in the embodiment. For example, the acquisition unit 1201 can be used to acquire a first mask image of the image to be processed, the mask processing unit 1202 is used to perform a first process on the first mask image, the segmentation unit 1203 is used to segment the foreground and background of the image to be processed to obtain a first foreground image and a first background image, the personalized processing unit 1204 performs personalized processing on the first foreground image and / or the first background image in accordance with user needs to obtain a second foreground image and a second background image, and the fusion unit 1205 is used to perform fusion processing on the second foreground image and the second background image.

[0101] The present application also provides an electronic device, such as Fig.13As shown, the electronic device 1300 may include a processor 1310. Optionally, the electronic device 1300 may also include a memory 1320. The memory 1320 may be disposed inside the electronic device 1300 or outside the electronic device 1300. Fig.12 The acquisition unit 1201, mask processing unit 1202, segmentation unit 1203, personalized processing unit 1204 and fusion unit 1205 shown in the figure can all be implemented by the processor 1310. Optionally, the electronic device 1300 may also include a display screen 1330 and a camera 1340. Among them, the processor 1310 is coupled with the memory 1320, the display screen 1330 and the camera 1340. The coupling in the embodiment of the present application is an indirect coupling or communication connection between devices, units or modules, which can be electrical, mechanical or other forms, and is used for information exchange between devices, units or modules. It should be noted that in the embodiment of the present application, the display screen and the camera may be located on the electronic device or may not be located on the electronic device. For example, the display screen and / or the camera can be connected to the electronic device as an external device.

[0102] Specifically, the memory 1320 is used to store program instructions. The display screen 1330 is used to display a photo preview interface, which includes an image captured by the camera 1340. The processor 1310 is used to call the program instructions stored in the memory 1320 so that the electronic device 1300 executes Figure 5 The steps in the image processing method shown are executed by the electronic device.

[0103] It should be understood that the electronic device 1300 can be used to implement the following embodiments of the present application: Figure 5 The method and related features of the image processing method shown can be referred to above and will not be repeated here.

[0104] The embodiment of the present application also provides a computer-readable storage medium, which may include a memory, and the memory may store a program, and when the program is executed, the electronic device executes the above-mentioned Figure 5 All or part of the steps described in the method embodiments shown.

[0105] The present application also provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the above Figure 5 All or part of the steps described in the method embodiments shown.

[0106] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. For example, in the above embodiments, the first acquisition unit and the second acquisition unit may be the same unit or different units. The above integrated units may be implemented in the form of hardware or in the form of software functional units.

[0107] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0109] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive, nor is it intended to limit the present application to the disclosed precise form. According to the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to fully illustrate the principles of the present application and its practical application, so that other persons skilled in the art can make full use of the present application and various embodiments with various modifications suitable for the specific purpose contemplated.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire a first mask image for segmenting the foreground and background of an image to be processed, wherein the first mask image is a binary mask image, and the image to be processed is obtained by taking a photo or recording a video with a camera, or the image to be processed is an image in a gallery, or the image to be processed is an image to be processed in a photo editing software; Performing corrosion and / or expansion processing on the first mask image to obtain a third mask image; Performing a smoothing process on the third mask image, and performing a binarization process on the mask image obtained after the smoothing process according to a first threshold value to obtain a second mask image; Segmenting the foreground and background of the image to be processed based on the second mask image to obtain a first foreground image and a first background image of the image to be processed; The first foreground image and / or the first background image are personalized in accordance with user needs to obtain a second foreground image and a second background image, and a fusion weight is determined based on a mask image obtained after smoothing the second mask image, and the second foreground image and the second background image are fused according to the fusion weight.

2. The method according to claim 1, characterized in that Determining a first fusion weight according to a mask image obtained by smoothing the second mask image, and fusing the second foreground image and the second background image according to the first fusion weight, including: The second foreground image and the second background image are fused by the following formula to obtain a fused image; fusion i =mask i *foreground i +(1-mask i )*background i ; Among them, fusion i represents the pixel value of the i-th pixel of the fused image, mask i represents the pixel value of the i-th pixel point of the mask image obtained after the smoothing process, foreground i represents the pixel value of the i-th pixel of the second foreground image, background i Represents the pixel value of the i-th pixel of the second background image.

3. The method according to claim 1, characterized in that The corroding and / or dilating the first mask image comprises: After the first mask image is corroded, the corroded first mask image is then dilated, or, After the first mask image is expanded, the expanded first mask image is eroded.

4. The method according to claim 1, characterized in that The smoothing process is any one of the following processing methods: Mean filtering, median filtering, bilateral filtering or Gaussian filtering.

5. An electronic device, characterized in that: including a processor and a memory; The memory is used to store a first image and one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the electronic device can implement the method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a computer program, and when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 4.

7. A program product, characterized in that The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 4.

Images