Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium. The method comprises the steps that a first image is acquired, the first image comprises a main body area and a background area, and the main body area comprises at least one sub-area; a target model is called to carry out matting on a main body region in the first image to obtain a target main body region, the target model comprises at least one target sub-model, and each target sub-model in the at least one target sub-model and each sub-region in the at least one sub-region have a one-to-one correspondence relationship; the image matting module is used for matting the respective corresponding sub-regions; performing blurring processing on the background area in the first image to obtain a third image; and fusing a third image with the target main body region to obtain a target image. According to the technical scheme of the invention, the sub-region matting is carried out on the foreground region, so that the edge precision of each target foreground sub-region is higher, the edge precision of the target foreground region is higher, the definition of the foreground contour is improved, and the image quality is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device and storage medium. Background Art

[0002] With the development of terminal technology, users' demand for terminal shooting capabilities is gradually increasing.

[0003] Photographic portrait modes are usually designed to highlight the outlines and features of people, making them more distinct and eye-catching in photos. These modes usually use a larger aperture value, or use corresponding algorithms to simulate a larger aperture lens to produce a shallow depth of field effect, blur the background, and focus on the person.

[0004] However, when the outline features of the person are not clear enough, the quality of the photos taken in portrait mode will be affected. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device and storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising acquiring a first image, the first image comprising a main body area and a background area, the main body area comprising at least one sub-area; calling a target model to cut out the main body area in the first image to obtain a target main body area; the target model comprising at least one target sub-model, each target sub-model in the at least one target sub-model having a one-to-one correspondence with each sub-area in the at least one sub-area, and being used to cut out the corresponding sub-areas; blurring the background area in the first image to obtain a third image; and fusing the third image with the target main body area to obtain a target image.

[0007] In one embodiment, the main body area includes a first sub-area and a second sub-area; calling the target model to cut out the main body area in the first image to obtain the target main body area includes: calling the first target sub-model in the target model to cut out the first sub-area in the first image to obtain the first target main body sub-area, and removing the first sub-area in the first image to obtain the second image; calling the second target sub-model in the target model to cut out the second sub-area in the second image to obtain the second target main body sub-area; performing feature fusion based on the first target main body sub-area and the second target main body sub-area to obtain the target main body area.

[0008] In another embodiment, removing the first sub-region in the first image includes: performing a first processing on the first sub-region in the first image to obtain a second image, wherein the first processing includes cropping the first sub-region in the first image and scaling the cropped first sub-region.

[0009] In another embodiment, before the feature fusion based on the first target main sub-region and the second target main sub-region is performed, the method further includes: in response to performing a first processing on the first sub-region, performing a second processing on the second target main sub-region to obtain a third target main region, wherein the second processing is an inverse processing of the first processing.

[0010] In another embodiment, the target model is trained based on the following method: obtaining an initial training sample, the initial training sample includes a training main area and a training background area, the training main area includes a first training sub-area and a second training sub-area; identifying the first training sub-area based on a first identification frame, and determining the first training sub-area as a first training sample; removing the first training sub-area from the initial training sample to obtain a second training sample; training a basic model based on the first training sample and the second training sample respectively to obtain a first target sub-model and a second target sub-model, and determining the target model based on the first target sub-model and the second target sub-model.

[0011] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing device, comprising: an acquisition unit, for acquiring a first image, the first image comprising a main body area and a background area, the main body area comprising at least one sub-area; a processing unit, for calling a target model to cut out the main body area in the first image to obtain a target main body area; the target model comprises at least one target sub-model, each target sub-model in the at least one target sub-model has a one-to-one correspondence with each sub-area in the at least one sub-area, and is used to cut out the corresponding sub-areas; the processing unit is also used to blur the background area in the first image to obtain a third image, and merge the third image with the target main body area to obtain a target image.

[0012] In one embodiment, the processing unit removes the first sub-region in the first image based on the following method, including: performing a first processing on the first sub-region in the first image to obtain a second image, the first processing including cropping the first sub-region in the first image and scaling the cropped first sub-region.

[0013] In another embodiment, before performing feature fusion based on the first target body sub-region and the second target body sub-region, the processing unit is also used to: in response to the first processing on the first sub-region, perform a second processing on the second target body sub-region to obtain a third target body region, wherein the second processing is the inverse processing of the first processing.

[0014] In another embodiment, the processing unit is also used to train the target model in the following manner: obtain an initial training sample, the initial training sample includes a training main area and a training background area, the training main area includes a first training sub-area and a second training sub-area; identify the first training sub-area based on a first identification frame, and determine the first training sub-area as a first training sample; remove the first training sub-area from the initial training sample to obtain a second training sample; train the basic model based on the first training sample and the second training sample respectively to obtain a first target sub-model and a second target sub-model, and determine the target model based on the first target sub-model and the second target sub-model.

[0015] According to a third aspect of an embodiment of the present disclosure, an image processing device is provided, comprising: a memory for storing processor executable commands; wherein the processor is configured to: execute the image processing method described in the first aspect or any one of the implementations of the first aspect.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a device, the device is enabled to execute the image processing method described in the first aspect or any one of the embodiments of the first aspect.

[0017] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: The present disclosure proposes an image processing method, which uses a target model to perform a cutout process on each sub-region of a subject region to obtain a target subject sub-region corresponding to each sub-region, and determines a target subject region based on the target subject sub-region, wherein the edge accuracy of the target subject region is higher than the edge accuracy of the subject region. By performing a cutout process on the subject region in different regions, the edge accuracy of each target subject sub-region can be made higher, thereby making the edge accuracy of the target subject region higher, thereby improving the clarity of the subject contour and improving the picture quality.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] Figure 1 The figure is a schematic diagram of a photographic portrait mode according to an exemplary embodiment.

[0021] Figure 2 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0022] Figure 3 The present invention is a flowchart of subject cutout in an image processing method according to an exemplary embodiment.

[0023] Figure 4 The present invention is a flowchart of performing main body cutout on a first sub-region in an image processing method according to an exemplary embodiment.

[0024] Figure 5 The present invention is a flowchart of performing main cutout on a second sub-region in an image processing method according to an exemplary embodiment.

[0025] Figure 6 The figure is a flowchart of a target model training according to an exemplary embodiment.

[0026] Figure 7 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0027] Figure 8 The figure is a schematic diagram for comparing effects of an image processing method according to an exemplary embodiment.

[0028] Fig. 9 The figure is a flow chart of a method for cropping and cutting out a portrait according to an exemplary embodiment.

[0029] Fig.10 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0030] Fig.11 The invention is a block diagram of a device for image processing according to an exemplary embodiment.

[0031] Fig.12 The invention is a block diagram of a device for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0033] With the development of terminal technology, users' demands for taking photos are becoming more and more diverse, which in turn prompts terminals to develop more diverse photo modes to meet users' photo needs. Portrait mode is a common photo mode. For example, Figure 1 FIG. 1 is a schematic diagram showing a photographic portrait mode according to an exemplary embodiment. Figure 1 As shown, in the case of the photographic portrait mode, the terminal camera will focus on person A, so that the scene B as the background object in the real image will be blurred, and the scene C in the photographic portrait image will be obtained. Among them, the portrait A can be called the subject, and the scene B can be called the background.

[0034] In some embodiments, the algorithm of the photography portrait mode is generally implemented based on applying different levels of image blur to clear images at different depths.

[0035] For ease of understanding and description, in some of the following embodiments, when there are human features in the captured image, "portrait", "subject" and "foreground" can be replaced with each other.

[0036] It can be understood that the “subject” is merely an illustrative example of the “human portrait”, and the subject in the embodiments of the present disclosure may also include other features besides the “human portrait”.

[0037] It is understandable that in the photography portrait mode, the clarity of the person's outline will affect the quality of the final generated image. For example, when the task outline is not clear enough, there will be no obvious boundary with the blurred background, or the edge details of the portrait terminal (such as human hair and other features) cannot be reflected more clearly, which results in insufficient prominence of the portrait or missing edge details during visual viewing of the image. In the related art, the clarity of the person's outline is usually improved based on the matting algorithm.

[0038] In some embodiments, the matting algorithm is usually applied to the complete outline of the person in the original image. However, due to the performance of the terminal and the hardware memory overhead, it is difficult for the terminal to use a high-resolution matting algorithm, while using a lower-resolution matting algorithm will result in insufficient outline clarity.

[0039] Based on the above problems, the present disclosure proposes an image processing method, which uses a target model to perform a cutout process on each sub-region of the subject region to obtain a target subject sub-region corresponding to each sub-region, and determines a target subject region based on the target subject sub-region, wherein the edge accuracy of the target subject region is higher than the edge accuracy of the subject region. By performing a cutout process on the subject region in different regions, the edge accuracy of each target subject sub-region can be made higher, thereby making the edge accuracy of the target subject region higher, thereby improving the clarity of the subject contour and improving the picture quality.

[0040] It should be noted that, in order to facilitate understanding of the embodiments of the present disclosure, some terms used in the embodiments of the present disclosure are explained below to facilitate understanding by those skilled in the art.

[0041] Photography portrait mode: usually aims to highlight the outline and features of the subject (such as a person), making the subject more distinct and eye-catching in the photo. These modes usually use a larger aperture value, or use corresponding algorithms to simulate a larger aperture lens to produce a shallow depth of field effect, blur the background, and focus on the subject.

[0042] Cutout: It is a common image processing technique. It refers to the process of separating the main object in the image from the background. Usually, cutout is used to remove the unwanted background in the image, or place the main object in a different background to achieve some special effects or composite images.

[0043] Matting algorithm: is a technique used to separate the main object in an image from the background. Its goal is to accurately calculate the subject (foreground) / background transparency (alpha value) of each pixel in the image to achieve accurate segmentation effect.

[0044] The matting algorithm is mainly based on the color and transparency information of the image. Common matting algorithms include Laplacian-based algorithms, interpolation algorithms, and learning-based algorithms.

[0045] Edge accuracy: Edge accuracy refers to the accuracy and clarity of the boundary between the main object and the background. In image processing, edges are usually the transition area between the main object and the background. They can be the outline of an object, a change in shape, or a change in color.

[0046] Both the original image and the image after the cutout process have edge accuracy, but the accuracy level may be different. The edges of the original image are determined based on the shape and boundaries of the object inherent in the photo or drawing. However, when performing the cutout process, algorithms are usually required to estimate and extract the edges of the main object.

[0047] The edge accuracy of the image after the cutout processing depends on the cutout algorithm used and the processing parameter settings. Some advanced algorithms can capture the edges of the main object more accurately, producing clear and accurate boundaries. Some simple or low-level algorithms may cause edge blur, discontinuity or over-cropping problems.

[0048] Therefore, edge accuracy is an important indicator for evaluating the effect of a cutout algorithm. A good cutout algorithm should be able to accurately preserve the edges of the subject and produce a natural, seamless effect during synthesis or editing.

[0049] Inverse processing: In the field of image processing, it refers to the process of restoring an image that has been processed in some way back to the original image. When we process an image, such as applying filtering, contrast enhancement, denoising, cropping, and scaling, the appearance and characteristics of the image will be changed. Inverse processing aims to restore the appearance and characteristics of the original image as much as possible by reversing these operations.

[0050] Feature fusion: Feature fusion is a technique in image processing that aims to combine multiple feature maps from different feature extraction methods to obtain richer and more useful information. In computer vision and image analysis tasks, it is often necessary to extract different features from images, such as color, texture, shape, etc. However, a single feature often cannot fully describe the content of the image, so by fusing multiple features, the characteristics of the image can be more comprehensively represented.

[0051] Base model: In neural networks, a base model refers to a trained initial model, usually a simpler model that provides a starting point for more complex models. The base model can be a basic neural network structure, such as a simple fully connected layer network, or a pre-trained model, such as ResNet or BERT on ImageNet.

[0052] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 As shown, the image processing method includes the following steps.

[0053] In step S11, a first image is acquired, the first image includes a main body area and a background area, and the main body area includes at least one sub-area.

[0054] In step S12, the target model is called to perform cutout on the main area in the first image to obtain the target main area. The target model includes at least one target sub-model. Each target sub-model in the at least one target sub-model has a one-to-one correspondence with each sub-area in the at least one sub-area, and is used to cutout the corresponding sub-areas.

[0055] In step S13, the background area in the first image is blurred to obtain a third image.

[0056] In step S14, the third image is fused with the target main area to obtain a target image.

[0057] The image processing method provided by the embodiment of the present disclosure obtains the target subject area by calling the target model to perform cutout processing on each sub-area of ​​the subject area in the first image. Since each sub-model corresponds to the cutout processing of a sub-area, compared with the cutout processing uniformly performed on the entire portrait area in the related art, the focus of the model cutout processing is improved, so that the cutout processing effect is better, and the edge accuracy of the cutout result corresponding to each sub-area is higher, thereby making the edge accuracy of the target subject area higher, so that the final target image display effect is better.

[0058] In some embodiments, the edge accuracy of the target body region is higher than the edge accuracy of the body region.

[0059] In some embodiments, the first image is an original image to be subjected to a cutout process.

[0060] In some embodiments, the background area may be other areas of the first image except the main body area.

[0061] Exemplary, continued Figure 1 In the embodiment of the present invention, the area where the portrait A is located can be understood as the portrait area, and can also be called the subject area. Figure 1 For portrait images, the area where the portrait A is located can also be called the foreground area. Then the area other than the area where the portrait A is located can be understood as the background area. For example, the sun and clouds in the picture.

[0062] In some embodiments, the target model is pre-trained by a base model.

[0063] In some embodiments, the target sub-model is pre-trained by the base model.

[0064] In some embodiments, the body region includes at least two sub-regions.

[0065] Exemplary, continued Figure 1In an embodiment, the sub-regions included in the main body region may be a region corresponding to the head (eg, may be called a head sub-region) and a region corresponding to the neck (eg, may be called a body sub-region).

[0066] In some embodiments, the main body region is composed of at least two sub-regions.

[0067] In some embodiments, the target body region is composed of at least two target body sub-regions.

[0068] Exemplary, continued Figure 1 In an embodiment, if the portrait A is cut out to obtain a target subject, the target subject area corresponding to the target subject may include an area corresponding to the head after cutout (for example, it may be called a head target sub-area) and an area corresponding to the neck (for example, it may be called a body target sub-area).

[0069] In some embodiments, each sub-region corresponds to independent feature information.

[0070] In some embodiments, the first sub-region corresponds to the first characteristic information, and the second sub-region corresponds to the second characteristic information.

[0071] Exemplary, continued Figure 1 In an embodiment, if the first sub-region is a head sub-region, the first feature information corresponding to the first sub-region is characterized as information of head features on the image, such as head contour information.

[0072] If the second sub-region corresponds to the region corresponding to the neck, the second feature information corresponding to the second sub-region is represented by information of body features on the image, for example, body contour information.

[0073] In some embodiments, the target portrait area is fused with the third image based on at least one of the following methods to obtain the target image, including:

[0074] Based on weighted feature summation (Weighted Feature Summation), based on feature averaging (Feature Averaging) and based on feature mapping (Feature Mapping), etc.

[0075] It can be understood that the main area includes at least two sub-areas, and the target main area includes at least two target main sub-areas. For the convenience of description, the above embodiment is described based on the example that the main area includes two sub-areas, and the target main area includes two target main sub-areas.

[0076] In addition, the main body area may also include a third sub-area, etc., and the target main body area may also include a third target main body sub-area, etc. If there is a third sub-area, the third sub-area corresponds to the third feature information. Based on this analogy, it will not be repeated here one by one.

[0077] In some embodiments, "edge accuracy", "edge accuracy", "edge precision", "outline accuracy", "outline accuracy", "outline accuracy", "edge clarity" and "outline clarity" can be used interchangeably and can be interpreted as the obviousness of the edge of an area, but are not limited to this.

[0078] In some embodiments, "obtain", "obtain", "get", "receive", "transmit", "bidirectional transmission", "send and / or receive" can be interchangeable, which can be interpreted as receiving from other subjects, but is not limited to this.

[0079] In some embodiments, terms such as "certain", "preset", "preset", "set", "indicated", "some", "any", and "first" can be interchangeable, and "specific A", "preset A", "preset A", "set A", "indicated A", "some A", "any A", and "first A" can be interpreted as A pre-defined in a protocol, etc., or as A obtained through setting, configuration, or indication, etc., and can also be interpreted as specific A, some A, any A, or first A, etc., but is not limited to this.

[0080] In some embodiments, "model", "artificial intelligence model", "algorithm", etc. can be replaced with each other, but are not limited to this.

[0081] The image processing method involved in the embodiments of the present disclosure may include at least one of steps S11 to S14. For example, step S11 may be implemented as an independent embodiment, step S12 may be implemented as an independent embodiment, step S11+step S12 may be implemented as an independent embodiment, and step S11+step S12+step S13+step S14 may be implemented as independent embodiments, but are not limited thereto.

[0082] In some embodiments, step S11, step S12, and step S13 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0083] In some embodiments, step S11, step S12, and step S14 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0084] In some embodiments, step S11, step S13, and step S14 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0085] In some embodiments, step S12, step S13, and step S14 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0086] In some embodiments, calling at least one target sub-model to perform main cutout on at least one sub-region to obtain a target main region includes:

[0087] Calling the first target sub-model to perform main body cutout on the first sub-region to obtain the first target main body sub-region;

[0088] Calling the second target sub-model to perform main body cutout on the second sub-region to obtain the second target main body sub-region;

[0089] The features of the first target main sub-region and the second target main sub-region are fused to obtain the target main region.

[0090] The image processing method provided in the embodiment of the present disclosure realizes an end-to-end correspondence between each sub-model and the sub-region by calling the target sub-model to perform cutout processing on each sub-region, and can realize a more detailed cutout processing for the portrait region, so that the edge accuracy of each sub-target portrait region is higher. In some embodiments, the cutout processing actions of the first target sub-model and the second target sub-model for their respective corresponding sub-regions are independent, and can be performed simultaneously or sequentially according to a preset order.

[0091] It is understandable that if there are more than two target main body sub-regions, the features of all target main body regions are fused. And so on, which will not be described one by one here.

[0092] Figure 3 FIG. 1 is a flowchart of a subject cutout in an image processing method according to an exemplary embodiment. Figure 3 As shown, calling the target model to cut out the main body area in the first image to obtain the target main body area includes the following steps.

[0093] In step S21, the first target sub-model in the target model is called to perform cutout on the first sub-region in the first image to obtain the first target main sub-region, and the first sub-region is removed from the first image to obtain the second image.

[0094] In step S22, the second target sub-model in the target model is called to perform cutout on the second sub-region in the second image to obtain a second target main body sub-region.

[0095] In step S23, feature fusion is performed based on the first target body sub-region and the second target body sub-region to obtain the target body region.

[0096] The image processing method provided by the embodiment of the present disclosure realizes an end-to-end correspondence between each sub-model and the sub-region by calling the target sub-model to perform cutout processing on each sub-region respectively, so as to achieve more detailed cutout processing for the portrait region, and take out the second image of the first sub-region as the input of the second target sub-model, so as to eliminate the noise brought by the influencing factors of the input to the second target sub-model, thereby improving the accuracy of the main body cutout of the subsequent model of the first target sub-model.

[0097] In some embodiments, calling the first target sub-model to perform main body cutout on the first sub-region includes the following steps:

[0098] Inputting a first image into a first target sub-model to determine a first sub-region;

[0099] Perform main cutout on the first sub-region to obtain a first target main sub-region.

[0100] In some embodiments, the first model identifies the first sub-region based on the feature detection box.

[0101] In some embodiments, the feature detection frame is used to identify feature information corresponding to the first sub-region, and the feature detection frame is obtained based on pre-training.

[0102] In some embodiments, the first model performs subject cutout on the first sub-region identified based on the feature detection frame to obtain a first target subject sub-region.

[0103] In some embodiments, the first image is input to the second object sub-model to determine the second sub-region.

[0104] In some embodiments, the second image is input to the second object sub-model to determine the second sub-region.

[0105] In some embodiments, the second image is acquired based on:

[0106] Inputting a first image into a first target sub-model to determine a first sub-region;

[0107] The first sub-region is subjected to main body cutout and cropping to obtain a first target main body sub-region and a second image.

[0108] It can be understood that allowing the second target sub-model to determine the second sub-region based on the first image and performing main cutout on the second sub-region can be suitable for the scenario where the first target sub-model and the second target sub-model simultaneously perform main cutout processing on the first image. In this way, multi-channel cutout processing can improve the cutout efficiency.

[0109] It can be understood that allowing the second target sub-model to determine the second sub-region based on the second image and performing main cutout on the second sub-region can be applicable to the scenario where the first target sub-model and the second target sub-model perform main cutout processing on the first image in sequence according to a specific order. Compared with the solution of performing cutout processing at the same time, the overall system structure is more stable and has better robustness.

[0110] In some embodiments, the second target sub-model identifies the second sub-region based on the feature detection box.

[0111] In some embodiments, the feature detection frame is used to identify feature information corresponding to the second sub-region, and the feature detection frame is obtained based on pre-training.

[0112] In some embodiments, the second target sub-model performs main body cutout on the second sub-region identified based on the feature detection frame to obtain the second target main body sub-region.

[0113] Exemplary, continued Figure 1 In the embodiment, the first target sub-model is called to perform a cutout process on the head region to obtain a head sub-target portrait region, and the second target sub-model is called to perform a cutout process on the neck region to obtain a body sub-target portrait region.

[0114] The second target sub-model may recognize the neck area based on the original image, or may recognize the neck area based on the cropped image after the head features are cropped by the first target sub-model.

[0115] The image processing method involved in the embodiment of the present disclosure may include at least one of steps S21 to S23. For example, step S21 may be implemented as an independent embodiment, step S22 may be implemented as an independent embodiment, and step S21+step S22+step S23 may be implemented as independent embodiments, but are not limited thereto.

[0116] In some embodiments, step S21 and step S22 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0117] In some embodiments, step S21 and step S23 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0118] In some embodiments, step S22 and step S23 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0119] Figure 4 is a flowchart showing a method of removing a first sub-region according to an exemplary embodiment. Figure 4 As shown, removing the first sub-region in the first image includes the following steps.

[0120] In step S31, a first image is acquired.

[0121] In step S32, a first process is performed on the first sub-region in the first image to obtain a second image, wherein the first process includes cutting out the first sub-region in the first image and scaling the cut out first sub-region.

[0122] The image processing method provided in the embodiment of the present disclosure can improve the quality of input data of the second target sub-model by performing a first processing on the first sub-region in the first image, further meet the input requirements of the second target sub-model, and thus improve the quality of the cutout processing results of the second target sub-model.

[0123] In some embodiments, the first process may also be an empty process action or omitted.

[0124] The image processing method involved in the embodiment of the present disclosure may include at least one of step S31 to step S32. For example, step S31 may be implemented as an independent embodiment, step S32 may be implemented as an independent embodiment, and step S31+step S32 may be implemented as independent embodiments, but it is not limited thereto.

[0125] In some embodiments, step S31 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0126] In some embodiments, step S32 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0127] Figure 5 is a flowchart of an image processing method according to an exemplary embodiment. Figure 5 As shown, before performing feature fusion based on the first target main sub-region and the second target main sub-region, the following steps are also included.

[0128] In step S41 , it is determined that the first sub-region has been subjected to the first processing.

[0129] In step S42, a second process is performed on the second target body sub-region to obtain a third target body region, wherein the second process is an inverse process of the first process.

[0130] In some embodiments, where the first process is a null process, the second process is a null process.

[0131] In some embodiments, where the first process is omitted, the second process is omitted.

[0132] In some embodiments, when the first processing is present, the first target body sub-region is merged with the third target body region to obtain the target body region.

[0133] In some embodiments, when the first process is an empty process or is omitted, the first target body sub-region and the second target body sub-region are merged to obtain the target body region.

[0134] Figure 6 is a flowchart of a target model training according to an exemplary embodiment. Figure 6 As shown, the following steps are included.

[0135] In step S61, an initial training sample is obtained, the initial training sample includes a training main area and a training background area, and the training main area includes a first training sub-area and a second training sub-area.

[0136] In step S62, a first training sub-region is identified based on the first identification frame, and the first training sub-region is determined as a first training sample.

[0137] In step S63, the first training sub-region is removed from the initial training sample to obtain a second training sample.

[0138] In step S64, the basic model is trained based on the first training sample and the second training sample to obtain a first target sub-model and a second target sub-model, and the target model is determined based on the first target sub-model and the second target sub-model.

[0139] The image processing method provided in the embodiment of the present disclosure trains each basic model using a training set with independent sub-region features to obtain a sub-model corresponding to each sub-region.

[0140] In some embodiments, before obtaining at least one training set, the method further includes: preprocessing the training set.

[0141] In some embodiments, preprocessing the training set includes: cropping or scaling the specified features in the training set. In this way, the preprocessed training set facilitates the basic model to recognize the specified features.

[0142] The image processing method involved in the embodiment of the present disclosure may include at least one of steps S61 to S64. For example, step S61 may be implemented as an independent embodiment, step S62 may be implemented as an independent embodiment, and step S61+step S62+step S63+step S64 may be implemented as independent embodiments, but are not limited thereto.

[0143] In some embodiments, steps S61, S62, and S63 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0144] In some embodiments, steps S61, S63, and S64 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0145] In some embodiments, steps S61, S62, and S64 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0146] In some embodiments, steps S62, S63, and S64 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0147] Figure 7 is a flowchart of an image processing method according to an exemplary embodiment. Figure 7 As shown, the following steps are included.

[0148] In step S71, training data is acquired and preprocessed.

[0149] In step S72, neural network model training is performed based on the training data.

[0150] In step S73, the portrait image is cropped and matted to obtain a matted image.

[0151] In step S74, the original image is processed in portrait mode, and the processed image is merged with the cutout image to obtain a target image.

[0152] The image processing method provided in the embodiment of the present disclosure can improve the edge accuracy of the portrait cutout, thereby improving the photo effect. Figure 8 As shown, Figure 8 FIG. 1 is a schematic diagram showing the effect comparison of an image processing method according to an exemplary embodiment. Figure 8In the comparison results, it can be clearly observed that the image processing performed by this method significantly improves the clarity of the portrait contour accuracy (such as hair details).

[0153] In some embodiments, obtaining training data and preprocessing the training data includes: obtaining training data and processing a portrait data set in the training data to obtain a face frame area. It can be understood that the training data is image data containing portraits.

[0154] In some embodiments, the face frame area is obtained based on a face detection algorithm.

[0155] Exemplarily, the coordinate information and size information of the initial face frame are obtained based on the face detection algorithm. For example, the coordinates (x, y) of a vertex of the initial face frame, the width w of the initial face frame, and the height h of the initial face frame are obtained (assuming that the face frame is a rectangle, where x, y, w, and h are all numbers greater than 0). The coordinate information and size information of the initial face frame are adaptively adjusted (for example, the size is expanded) to cover the complete head area, thereby obtaining the face frame area.

[0156] It is understandable that the coordinate information of the face frame is expanded in size to obtain head features. The expanded face frame can cover more head features to avoid missing head features.

[0157] In some embodiments, the face frame area in each training data is cropped to obtain face training data.

[0158] In some embodiments, performing neural network training model training based on training data includes:

[0159] A neural network is trained based on the training data to obtain a first model, and a neural network is trained based on the face training data to obtain a second model.

[0160] It is understandable that the second model can perform designated feature recognition on the face region, and the first model can perform designated feature recognition on regions other than the face region.

[0161] Fig. 9 FIG. 1 is a flowchart of a method for cropping a portrait according to an exemplary embodiment. Fig. 9 As shown, the head region of the portrait image is cropped, and the cropped head region and the remaining region (e.g., the human body region) are respectively put into the trained neural network (e.g., the human body region is put into neural network 1, and the head region is put into neural network 2), and the image is cut out to obtain the image cutout results of each region (e.g., the human body image cutout result and the head region image cutout result). The image cutout results are then fused to obtain the final image cutout result of the portrait.

[0162] In some embodiments, the final portrait cutout result I Final It can be expressed based on the following formula:

[0163] I Final =α*I+(1-α)*I bokeh

[0164] Among them, α is the weight solved based on the cutout processing, I is the original image (the original image includes the portrait area and the background area), I bokeh This is an image with blurred background using the portrait mode.

[0165] Based on the same concept, an embodiment of the present disclosure also provides an image processing device.

[0166] Fig.10 FIG. 1 is a block diagram of an image processing apparatus 100 according to an exemplary embodiment. Fig.10 The device includes an acquisition unit 101 and a processing unit 102.

[0167] The acquisition unit 101 is used to acquire a first image, where the first image includes a main area and a background area, and the main area includes at least one sub-area.

[0168] The processing unit 102 is used to call the target model to cut out the main area in the first image to obtain the target main area; the target model includes at least one target sub-model, each target sub-model in the at least one target sub-model has a one-to-one correspondence with each sub-area in the at least one sub-area, and is used to cut out the corresponding sub-areas.

[0169] In some embodiments, the processing unit 102 obtains the target body area based on the following method, including: calling the first target sub-model in the target model to cut out the first sub-region in the first image to obtain the first target body sub-region, and removing the first sub-region in the first image to obtain the second image; calling the second target sub-model in the target model to cut out the second sub-region in the second image to obtain the second target body sub-region; performing feature fusion based on the first target body sub-region and the second target body sub-region to obtain the target body area.

[0170] In some embodiments, the processing unit 102 removes the first sub-region in the first image based on the following method, including: performing a first processing on the first sub-region in the first image to obtain a second image, the first processing including cropping the first sub-region in the first image, and scaling the cropped first sub-region.

[0171] In some embodiments, before performing feature fusion based on the first target body sub-region and the second target body sub-region, the processing unit 102 is also used to: in response to the first processing on the first sub-region, perform a second processing on the second target body sub-region to obtain a third target body region, wherein the second processing is the inverse processing of the first processing.

[0172] In some embodiments, the processing unit 102 obtains the target image based on the following methods, including: blurring the background area to obtain a third image; and fusing the target main area with the third image to obtain the target image.

[0173] In some embodiments, the processing unit 102 is also used to train a target model in the following manner: obtain an initial training sample, the initial training sample includes a training main area and a training background area, the training main area includes a first training sub-area and a second training sub-area; identify the first training sub-area based on a first identification frame, and determine the first training sub-area as a first training sample; remove the first training sub-area from the initial training sample to obtain a second training sample; train the basic model based on the first training sample and the second training sample respectively to obtain a first target sub-model and a second target sub-model, and determine the target model based on the first target sub-model and the second target sub-model.

[0174] It is understandable that in order to realize the above functions, the image processing device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0175] Fig.11 2 is a block diagram of an apparatus 200 for image processing according to an exemplary embodiment. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0176] Reference Fig.11 The image processing device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0177] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0178] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0179] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0180] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0181] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0182] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0183] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0184] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0185] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0186] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0187] Fig.12 is a block diagram of a device 300 for image processing according to an exemplary embodiment. For example, the device 300 may be provided as a server. Fig.12 The apparatus 300 includes a processing component 322, which further includes one or more processors, and a memory resource represented by a memory 332 for storing instructions executable by the processing component 322, such as an application. The application stored in the memory 332 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 322 is configured to execute instructions to perform the above-mentioned screen display method.

[0188] The device 300 may also include a power supply component 326 configured to perform power management of the device 300, a wired or wireless network interface 350 configured to connect the device 300 to a network, and an input / output (I / O) interface 358. The device 300 may operate based on an operating system stored in the memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0189] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0190] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0191] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.

[0192] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0193] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0194] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0195] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first image, the first image including a main area and a background area, the main area including at least one sub-area; Calling a target model to cut out a main body region in the first image to obtain a target main body region, wherein the target model includes at least one target sub-model, each target sub-model in the at least one target sub-model has a one-to-one correspondence with each sub-region in the at least one sub-region, and is used to cut out the corresponding sub-regions; Performing blur processing on the background area in the first image to obtain a third image; The third image is fused with the target main area to obtain a target image.

2. The image processing method according to claim 1, characterized in that: The main body area includes a first sub-area and a second sub-area; The calling the target model to cut out the main body area in the first image to obtain the target main body area includes: Calling a first target sub-model in the target model to cut out a first sub-region in the first image to obtain a first target main sub-region, and removing the first sub-region from the first image to obtain a second image; Calling a second target sub-model in the target model to cut out a second sub-region in the second image to obtain a second target main sub-region; The target body region is obtained by performing feature fusion based on the first target body sub-region and the second target body sub-region.

3. The image processing method according to claim 2, characterized in that: The removing the first sub-region from the first image includes: A first process is performed on a first sub-region in the first image to obtain a second image, wherein the first process includes cutting out a first sub-region in the first image and scaling the cut out first sub-region.

4. The image processing method according to claim 2, characterized in that: Before the feature fusion is performed based on the first target main sub-region and the second target main sub-region, the method further includes: In response to performing a first process on the first sub-region, performing a second process on the second target body sub-region to obtain a third target body region, wherein the second process is an inverse process of the first process.

5. The image processing method according to claim 1 or 2, characterized in that: The target model is trained based on the following method: Acquire an initial training sample, wherein the initial training sample includes a training main area and a training background area, and the training main area includes a first training sub-area and a second training sub-area; Identify the first training sub-region based on the first identification frame, and determine the first training sub-region as a first training sample; Removing the first training sub-region from the initial training sample to obtain a second training sample; The basic model is trained based on the first training sample and the second training sample respectively to obtain a first target sub-model and a second target sub-model, and the target model is determined based on the first target sub-model and the second target sub-model.

6. An image processing device, characterized in that: include: An acquisition unit, configured to acquire a first image, wherein the first image includes a main body area and a background area, and the main body area includes at least one sub-area; a processing unit, configured to call a target model to perform cutout on a main body region in the first image to obtain a target main body region, wherein the target model includes at least one target sub-model, each target sub-model in the at least one target sub-model has a one-to-one correspondence with each sub-region in the at least one sub-region, and is configured to perform cutout on the respective corresponding sub-regions; The processing unit is further used to perform blurring processing on the background area in the first image to obtain a third image, and merge the third image with the target subject area to obtain a target image.

7. The image processing device according to claim 6, characterized in that: The processing unit obtains the target main area based on the following method, including: Calling a first target sub-model in the target model to cut out a first sub-region in the first image to obtain a first target main sub-region, and removing the first sub-region from the first image to obtain a second image; Calling a second target sub-model in the target model to cut out a second sub-region in the second image to obtain a second target main sub-region; The target body region is obtained by performing feature fusion based on the first target body sub-region and the second target body sub-region.

8. The image processing device according to claim 7, characterized in that: The processing unit removes the first sub-region in the first image based on the following method, including: A first process is performed on a first sub-region in the first image to obtain a second image, wherein the first process includes cutting out a first sub-region in the first image and scaling the cut out first sub-region.

9. The image processing device according to claim 7, characterized in that: Before the processing unit performs feature fusion based on the first target main sub-region and the second target main sub-region, the processing unit is further used to: In response to performing a first process on the first sub-region, performing a second process on the second target body sub-region to obtain a third target body region, wherein the second process is an inverse process of the first process.

10. The image processing device according to claim 6 or 7, characterized in that: The processing unit is also used to train the target model in the following manner: Acquire an initial training sample, wherein the initial training sample includes a training main area and a training background area, and the training main area includes a first training sub-area and a second training sub-area; Identify the first training sub-region based on the first identification frame, and determine the first training sub-region as a first training sample; Removing the first training sub-region from the initial training sample to obtain a second training sample; The basic model is trained based on the first training sample and the second training sample respectively to obtain a first target sub-model and a second target sub-model, and the target model is determined based on the first target sub-model and the second target sub-model.

11. An image processing device, characterized in that: include: a memory for storing processor executable instructions; Wherein, the processor is configured to: execute the image processing method described in any one of claims 1 to 5.

12. A storage medium storing instructions, characterized in that: When the instruction is executed on a device, the device executes the image processing method according to any one of claims 1 to 5.

Images