Image processing method and device, electronic equipment, storage medium and computer program product

Abstract

The invention discloses an image processing method and device, electronic equipment, a storage medium and a computer program product, and belongs to the technical field of image.The method comprises the steps that a first image and a second image are input into a first sectional drawing model, and a first feature map is obtained based on the first image and the second image through the first sectional drawing model, the second image is an image obtained based on image elements in the first image; processing the first feature map through a first matting model to obtain at least one of a detail feature map and an edge feature map; outputting a first cutout image based on at least one of the detail feature map and the edge feature map and the first feature map through a first cutout model; wherein the first matting model is obtained by training based on detail features and edge features of the image.

Description

Technical Field

[0001] The present application belongs to the field of image technology, and specifically relates to an image processing method, device, electronic device, storage medium and computer program product. Background Art

[0002] Generally, after a user captures an image through a mobile phone, the user may trigger the mobile phone to perform image processing on the captured image.

[0003] For example, if the user finds that the position of the subject in the captured image does not meet his or her needs, the user can use the cutout function of the image processing software in the mobile phone to cut out the subject, and then the user can move the cutout image to the required position in the captured image, so that the processed image can meet the user's needs.

[0004] However, in the above method, due to the limited cutout accuracy of the cutout function in the image processing software, the display effect of the cutout image may not meet the needs of the user, thus resulting in poor image quality of the cutout image obtained by the electronic device. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device, storage medium and computer program product, which can improve the image quality of the cutout image obtained by the electronic device.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, which includes: inputting a first image and a second image into a first cutout model, and obtaining a first feature map based on the first image and the second image through the first cutout model, wherein the second image is an image obtained based on image elements in the first image; processing the first feature map through the first cutout model to obtain at least one of a detail feature map and an edge feature map; outputting a first cutout image based on at least one of the detail feature map and the edge feature map and the first feature map through the first cutout model; wherein the first cutout model is trained based on the detail features and edge features of the image.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, which includes: an acquisition module, a processing module and an output module; the acquisition module is used to input a first image and a second image into a first cutout model, and obtain a first feature map based on the first image and the second image through the first cutout model, wherein the second image is an image obtained based on image elements in the first image; the processing module is used to process the first feature map through the first cutout model to obtain at least one of a detail feature map and an edge feature map; the output module is used to output a first cutout image based on at least one of the detail feature map and the edge feature map, and the first feature map through the first cutout model; wherein the first cutout model is trained based on the detail features and edge features of the image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program / program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, the electronic device may input the first image and the second image into a first cutout model to obtain a first feature map based on the first image and the second image through the first cutout model, where the second image is an image obtained based on image elements in the first image; then, the electronic device may process the first feature map through the first cutout model to obtain at least one of a detail feature map and an edge feature map; and output a first cutout image based on at least one of the detail feature map and the edge feature map, and the first feature map through the first cutout model; wherein the first cutout model is trained based on the detail features and edge features of the image. In this scheme, since the second image is an image obtained based on the image elements that need to be cut out in the first image, the second image includes the image elements that need to be cut out. Therefore, the electronic device can process based on the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. That is to say, the first cutout image output by the first cutout model can accurately retain at least one of the detail image content and the edge image content in the image elements that the user needs to cut out in the first image based on the second image, so that the first cutout image can meet the user's demand for the fineness of the cutout image, thereby improving the image quality of the cutout image obtained by the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flow charts of an image processing method provided in an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of an example of a cutout image provided in an embodiment of the present application;

[0015] Figure 3 It is one of the structural diagrams of a first cutout model provided in an embodiment of the present application;

[0016] Figure 4A This is the second structural diagram of a first cutout model provided in an embodiment of the present application;

[0017] Figure 4B This is a third structural diagram of a first cutout model provided in an embodiment of the present application;

[0018] Figure 5A This is one of the example schematic diagrams of a mask image provided in an embodiment of the present application;

[0019] Figure 5B This is one of the example schematic diagrams of a mask image provided in an embodiment of the present application;

[0020] Figure 6 This is a fourth structural diagram of a first cutout model provided in an embodiment of the present application;

[0021] Figure 7 This is a fifth structural diagram of a first cutout model provided in an embodiment of the present application;

[0022] Figure 8 This is the second flowchart of an image processing method provided in an embodiment of the present application;

[0023] Fig. 9A This is one of the example schematic diagrams of a mobile phone displaying a cutout image provided by an embodiment of the present application;

[0024] Fig. 9B This is a second example schematic diagram of a mobile phone displaying a cutout image provided by an embodiment of the present application;

[0025] Fig. 9C This is a third example schematic diagram of a mobile phone displaying a cutout image provided by an embodiment of the present application;

[0026] Fig. 10A This is a fourth example schematic diagram of a mobile phone displaying a cutout image provided by an embodiment of the present application;

[0027] Fig. 10B This is a fifth example schematic diagram of a mobile phone displaying a cutout image provided by an embodiment of the present application;

[0028] Fig.11 This is the third flowchart of an image processing method provided in an embodiment of the present application;

[0029] Fig.12 This is the sixth example schematic diagram of a mobile phone displaying a cutout image provided by an embodiment of the present application;

[0030] Fig.13A This is the seventh example schematic diagram of a mobile phone displaying a cutout image provided by an embodiment of the present application;

[0031] Fig. 13B This is an eighth example schematic diagram of a mobile phone displaying a cutout image provided by an embodiment of the present application;

[0032] Fig.14 It is a flow chart of a method for training a cutout model provided in an embodiment of the present application;

[0033] Fig.15 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0034] Fig.16 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0035] Fig.17 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0037] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0038] The terms "at least one (item)", "at least one of" and the like in the specification of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0039] The following is an explanation of the professional terms involved in the embodiments of the present application:

[0040] Secondary composition: The position of the subject contained in the image stored in the electronic device is moved to obtain a composition picture of the new subject position.

[0041] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0042] The image processing method provided in the embodiment of the present application can be applied to scenarios where secondary composition of the image is required.

[0043] The image processing method provided in the embodiments of the present application is exemplarily described below using some specific scenarios as examples.

[0044] Scenario 1: Secondary composition of the characters in the image

[0045] If the user wants to recompose an image, such as adjusting the position of person 1 in image 1 relative to the background building, the user can trigger the electronic device to input image 1 and the cutout image corresponding to image 1 into the first cutout model, so that the first cutout model outputs a first cutout image with detail features and edge features corresponding to person 1 based on the input image.

[0046] Scenario 2: Secondary composition of objects in the image

[0047] If the user wants to recompose an image, such as adjusting an object in image 2, such as the display position and size of object 2, the user can trigger the electronic device to input image 2 and the cutout image corresponding to image 2 into the first cutout model, so that the first cutout model outputs a first cutout image with detail features and edge features corresponding to object 2 based on the input image.

[0048] It should be noted that the above-mentioned scenarios 1 and 2 are merely illustrative examples of some scenarios in which the embodiments of the present application may be applied. In actual implementation, the embodiments of the present application can also be applied to more possible scenarios that require adjustment of parameters such as the display size and display position of image elements contained in the image, and the embodiments of the present application are not limited here.

[0049] Based on the above-mentioned scenario applied by the embodiments of the present application, in the image processing method provided by the embodiments of the present application, the electronic device can input the first image and the second image into the first cutout model to obtain a first feature map based on the first image and the second image through the first cutout model, and the second image is an image obtained based on the image elements in the first image; then, the electronic device can process the first feature map through the first cutout model to obtain at least one of a detail feature map and an edge feature map; and output the first cutout image through the first cutout model based on at least one of the detail feature map and the edge feature map, and the first feature map; wherein the first cutout model is trained based on the detail features and edge features of the image. In this scheme, since the second image is an image obtained based on the image elements that need to be cut out in the first image, the second image includes the image elements that need to be cut out. Therefore, the electronic device can process based on the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. That is to say, the first cutout image output by the first cutout model can accurately retain at least one of the detail image content and the edge image content in the image elements that the user needs to cut out in the first image based on the second image, so that the first cutout image can meet the user's demand for the fineness of the cutout image, thereby improving the image quality of the cutout image obtained by the electronic device.

[0050] The image processing method provided in the embodiment of the present application is executed by an image processing device, which may be an electronic device, or a functional module or entity in an electronic device, and the embodiment of the present application does not limit this. The image processing method provided in the embodiment of the present application will be exemplarily described below using an electronic device as an example.

[0051] The present application provides an image processing method. Figure 1 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present application. Figure 1 As shown, the image processing method provided in the embodiment of the present application may include the following steps 201 to 203.

[0052] Step 201: The electronic device inputs a first image and a second image into a first cutout model, and obtains a first feature map based on the first image and the second image through the first cutout model.

[0053] In the embodiment of the present application, the second image may be an image obtained based on image elements in the first image.

[0054] In some embodiments of the present application, the image elements included in the second image may be determined according to a user's requirement for cutting out the first image.

[0055] In some embodiments of the present application, the second image may have the same image size as the first image.

[0056] In some embodiments of the present application, the second image may be a mask image corresponding to the first image.

[0057] It should be noted that the above mask may be a simple black and white mask, or a complex mask based on the layer content included in the first image.

[0058] In some embodiments of the present application, the mask may be used to limit a display area for image elements that require a cutout process to be performed from the first image.

[0059] For example, assuming that the first image includes a head image of a male user wearing glasses, and the user wants to obtain the head image through the electronic device, the user can first trigger the electronic device to perform a cutout process based on the head image, so that the electronic device outputs the following image: Figure 2 The black and white mask image corresponding to the head image shown is the second image. The white area in the black and white mask image is the image area that the user wants to cut out, that is, the head image of the male user wearing glasses; the black area is the image area that the user does not need to cut out, that is, the background area in the first image.

[0060] In the embodiment of the present application, the first cutout model is trained based on the detail features and edge features of the image.

[0061] In some embodiments of the present application, when an electronic device inputs a first image and a second image into a first cutout model, the first cutout model may first merge the first image and the second image into one image by superimposing image channels, and then process the merged image to obtain a first feature map.

[0062] In some embodiments of the present application, the first cutout model may include an input convolution layer and N convolution modules.

[0063] In some embodiments of the present application, when the electronic device inputs the first image and the second image into the first cutout model, the electronic device may first downsample the image merged by the first image and the second image through an input convolution layer to reduce the resolution of the image.

[0064] For example, assuming that the resolution of the first image and the second image is 1024*1024, after downsampling processing is performed on the input convolution layer, the resolution of the output feature map can be reduced to 512*512.

[0065] In some embodiments of the present application, the first image and the second image can perform matrix calculation based on Feature=w0x+b0 in the input convolution layer to extract features in the input image, thereby outputting a feature map corresponding to the input convolution layer. Wherein, x can be image data obtained from the first image and the second image, w0 can be a weight value corresponding to the input convolution layer, and b0 can be a bias value corresponding to the input convolution layer.

[0066] In this way, the electronic device downsamples the image input to the model to adjust the size of the input image, thereby reducing the amount of computational work for image processing in the model and the number of parameters used in model calculation, while increasing the robustness and generalization ability of the model.

[0067] In some embodiments of the present application, the electronic device may input the feature map output by the input convolution layer into the convolution module in the first cutout model to perform subsequent image processing on the image.

[0068] It should be noted that the number of convolution modules included in the above-mentioned first cutout model and the parameters of each convolution model can be determined according to actual needs, and this application does not limit them here.

[0069] In some embodiments of the present application, the "obtaining a first feature map based on the first image and the second image" in the above step 201 can be specifically implemented through the following steps 201a to 201c.

[0070] Step 201a: The electronic device downsamples the first image and the second image through the first convolution module in the first cutout model, and outputs a first feature vector.

[0071] Step 201b: The electronic device performs downsampling processing on the i-1th feature vector through the i-th convolution module in the first cutout model, and outputs the i-th feature vector.

[0072] Where i∈[2,N], i is an integer, and N is the number of convolutional modules.

[0073] Step 201c: determine the Nth eigenvector as the first eigenmap.

[0074] In some embodiments of the present application, the electronic device may perform N downsampling processes through N convolution modules on the basis of the feature map output by the above-mentioned input convolution layer to extract the semantic information of the first image and the second image, that is, the above-mentioned feature vector.

[0075] It should be noted that the semantic information of the above-mentioned image may include but is not limited to basic information such as the outline and category of the image elements contained in the image; it may also include information such as the attributes, positional relationship, and context of the image elements.

[0076] The attributes of the image element may include but are not limited to at least one of the following: color, shape, and size.

[0077] In some embodiments of the present application, during the process of downsampling the input image data by the convolution module, the electronic device may increase the number of convolution kernels and the number of output channels.

[0078] In this way, since each convolution kernel can extract a specific image feature and output a corresponding feature map, more types of features can be extracted from the image, enhancing the richness and diversity of feature representation, so that the model can better understand and recognize different elements and patterns in the image, thereby further avoiding the loss of image feature information.

[0079] For example, Figure 3As shown, if the above N=4, the electronic device can firstly downsample the first image and the second image in the first cutout model through the input convolution layer to obtain a feature map with a resolution of 512*512, and the feature map is obtained by combining images of 32 channels; secondly, the electronic device can downsample the feature map with a resolution of 512*512 output by the input convolution layer through the first convolution module, that is, convolution module 1, to obtain a feature map with a resolution of 256*256, and the feature map is obtained by combining images of 64 channels, that is, the first feature vector; then, the electronic device can downsample the feature map with a resolution of 256*256 output by the convolution module 1 through the second convolution module, that is, convolution module 2, to obtain a feature map with a resolution of 256*256. The feature map with a resolution of 128*128 is obtained by combining images of 128 channels, that is, the second feature vector; then, the electronic device can downsample the feature map with a resolution of 128*128 output by the convolution module 2 through the third convolution module, that is, the convolution module 3, to obtain a feature map with a resolution of 64*64, and the feature map is obtained by combining images of 256 channels, that is, the third feature vector; finally, the electronic device can downsample the feature map with a resolution of 64*64 output by the convolution module 3 through the fourth convolution module, that is, the convolution module 4, to obtain a feature map with a resolution of 32*32, and the feature map is obtained by combining images of 512 channels, that is, the fourth feature vector, which is also the first feature map.

[0080] In some embodiments of the present application, the first cutout model can be based on the i-th convolution module included, through Feature i =w i (w i-1 (…(w1x+b1)…)+b i-1 )+b i Perform matrix calculations to extract features from the input image, thereby outputting the first feature map. i can be the feature vector output by the i-th convolution module; x can be the image data received by the first convolution module, such as the feature map output by the input convolution layer; w i can be the weight value corresponding to the i-th convolution module; b i It can be the bias value corresponding to the i-th convolution module.

[0081] It should be noted that the weight value and bias value corresponding to each convolution module may be different. The weight value and bias value may be set as the value determined during the training process of the first cutout model, and the specific value is not limited in this application.

[0082] In this way, the electronic device can use the N convolution modules included in the first cutout model to perform N downsampling processing on the image received by the convolution module to adjust the size of the input image, thereby reducing the amount of computational processing on the image in the model and the number of parameters used in the model calculation, while increasing the robustness and generalization ability of the model.

[0083] Step 202: The electronic device processes the first feature map using a first cutout model to obtain at least one of a detail feature map and an edge feature map.

[0084] In some embodiments of the present application, the first cutout model may include N deconvolution modules.

[0085] In some embodiments of the present application, the electronic device may perform upsampling processing on the first feature map through a deconvolution module to improve the resolution of the first feature map, so that the subsequent first cutout image can output an image with the same resolution as the first image and the second image.

[0086] In some embodiments of the present application, the above-mentioned detail feature map can be understood as a feature map with image details obtained by finely segmenting the image elements that the user needs to cut out in the first image.

[0087] In some embodiments of the present application, the above-mentioned edge feature map can be understood as performing fine segmentation on the edge area of ​​the image element that the user needs to cut out in the first image, and the obtained feature map has a significantly improved display effect on the edge area.

[0088] It should be noted that the above-mentioned refined segmentation can be understood as a method with a segmentation accuracy higher than that of existing segmentation methods, for example, a segmentation method that can achieve accurate segmentation at the hair-level.

[0089] Among them, the above-mentioned existing segmentation method may include but is not limited to any of the following: an image segmentation method for segmenting through a single image of an RGB image; an image segmentation method for segmenting through a single image of an RGB image and a corresponding ternary image (trimap).

[0090] In some embodiments of the present application, in combination with the above steps 201a to 201c, the above step 202 of "processing the first feature map to obtain a detail feature map" can be specifically implemented through the following steps 202a1 to 202a3.

[0091] Step 202a1: The electronic device performs upsampling processing on the first feature map through the first deconvolution module in the first cutout model to obtain a first detail feature vector.

[0092] In some embodiments of the present application, the first deconvolution module in the first cutout model, namely, the deconvolution module 1, can be implemented by Perform matrix calculation to upsample the first feature map to obtain the first detail feature vector TransFeature1. may be the weight value corresponding to the deconvolution module 1; x may be the first feature map received by the deconvolution module 1; It can be the bias value corresponding to the first deconvolution module.

[0093] In some embodiments of the present application, the resolution of the image corresponding to the first detail feature vector may be greater than the resolution of the first feature map.

[0094] In some embodiments of the present application, the electronic device may perform a convolution operation using a transposed convolution kernel by performing an inverse operation on the convolution module to achieve upsampling of the first feature map, thereby obtaining a first detail feature vector.

[0095] In some embodiments of the present application, a deconvolution module in the first cutout model can be connected to a convolution module via a skip connection. Thus, the skip connection can first perform linear addition based on the detail feature vector output by the connected deconvolution module and the feature vector output by the convolution module, and then transmit the result to a subsequent deconvolution module for processing.

[0096] In this way, the first cutout model can enhance the image details in the feature map and improve the predicted details of the image in the feature map by fusing the feature maps of information at different semantic levels.

[0097] For example, in combination Figure 3 ,like Figure 4A As shown, after the convolution module 4 outputs the first feature map of 32*32*512, the electronic device can upsample the first feature map through the first deconvolution module, that is, the deconvolution module 1, to obtain a feature map with a resolution of 64*64, and the feature map is obtained by combining images of 64 channels, that is, the first detail feature vector. Then, the electronic device can linearly add TransFeature1 output by the deconvolution module 1 and Feature3 output by the convolution module 3 through skip1, that is, skip1=TransFeature1+Feature3, as the input of the second deconvolution module for subsequent processing.

[0098] Step 202a2: The electronic device performs upsampling processing on the Nj-th feature vector and the j-1-th detail feature vector through the j-th deconvolution module in the first cutout model, and outputs the j-th detail feature vector.

[0099] Among them, j∈[2,N].

[0100] In some embodiments of the present application, while the deconvolution module upsamples the input image data to increase the resolution of the feature map, the electronic device can compress the dimension of the output feature map to reduce the number of image channels, thereby facilitating the first cutout model to subsequently output a single cutout image to meet the user's application needs.

[0101] In some embodiments of the present application, methods for reducing the number of image channels may include but are not limited to any of the following: color space conversion, 1*1 convolution.

[0102] In some embodiments of the present application, the j-th deconvolution module in the first cutout model can be Perform matrix calculation to upsample the received feature vector to obtain the jth detail feature vector TransFeature j .in, It can be the weight value corresponding to the j-th deconvolution module; x can be the Nj-th feature vector and the j-1-th detail feature vector; It can be the bias value corresponding to the j-th deconvolution module.

[0103] For example, in combination Figure 4A , the second deconvolution module in the first cutout model, i.e., deconvolution module 2, can receive the image data transmitted by skip1, i.e., skip1 = TransFeature1 + Feature3, so deconvolution module 2 can pass Perform matrix calculation to upsample the feature vector corresponding to the received skip1 to obtain the second detail feature vector TransFeature2. It can be the weight value corresponding to the deconvolution module 2; It can be the offset value corresponding to the deconvolution module 2. In addition, the second detail feature vector TransFeature2 is a feature map with a resolution of 128*128, and the feature map is obtained by combining images of 32 channels.

[0104] Step 202a3: The electronic device determines the Nth detail feature vector as a detail feature map.

[0105] In some embodiments of the present application, the electronic device may determine the detail feature vector output by the last deconvolution module as a detail feature map.

[0106] For example, in combination Figure 4A ,like Figure 4BAs shown, the electronic device can first perform linear addition of TransFeature2 output by deconvolution module 2 and Feature2 output by convolution module 2 through skip2, that is, skip2=TransFeature2+Feature2, as the input of the third deconvolution module, that is, deconvolution module 3. Therefore, deconvolution module 3 can be Perform matrix calculation to upsample the feature vector corresponding to the received skip2 to obtain the third detail feature vector TransFeature3. It can be the weight value corresponding to the deconvolution module 3; It can be the offset value corresponding to the deconvolution module 3. Moreover, the third detail feature vector TransFeature3 is a feature map with a resolution of 256*256, and the feature map is obtained by combining images of 16 channels. Then, the electronic device can linearly add the TransFea3 output by the deconvolution module 3 and the Feature1 output by the convolution module 1 through skip3, that is, skip3=TransFeature3+Feature1. Therefore, the fourth deconvolution module in the first cutout model, that is, the deconvolution module 4, can receive the image data transmitted by skip3, so the deconvolution module 4 can be used Perform matrix calculation to upsample the received feature vector corresponding to skip3 to obtain the fourth detail feature vector TransFeature4. It can be the weight value corresponding to the deconvolution module 4; It can be the offset value corresponding to the deconvolution module 4. Moreover, the fourth detail feature vector TransFeature4 is a feature map with a resolution of 512*512, and the feature map is obtained by combining images of 16 channels. Therefore, the electronic device can determine the fourth detail feature vector TransFeature4 as a detail feature map.

[0107] In this way, the electronic device can improve the segmentation accuracy of the detail positions of the image elements to be cut out by the first cutout model through the convolution module and the deconvolution module, so that a cutout image with a higher degree of accuracy can be directly obtained later without the need for the user to manually draw the edge positions of the image elements, thereby reducing the steps that the user needs to perform manually. Furthermore, since feature fusion processing is performed on multiple feature maps during the operation of the deconvolution module, the details of each layer of feature output in the calculation process of the first cutout model can be enriched, thereby further improving the degree of refinement.

[0108] In some embodiments of the present application, after the above step 202a2, the image processing method provided by the embodiment of the present application further includes the following steps A1 and A2.

[0109] Step A1: The electronic device performs feature fusion on the j-1th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first cutout model to obtain the j / 2th edge feature vector.

[0110] Wherein, j is an even number.

[0111] In some embodiments of the present application, the above-mentioned edge attention module can enhance the segmentation details at the edges of the input image, so that the first cutout model can determine the central area through high-level features, so that the deconvolution module subsequent to the edge attention module in the first cutout model can focus on training the edge areas of image elements in the image to output a feature map with a fine edge effect.

[0112] In some embodiments of the present application, the above-mentioned feature fusion method may include but is not limited to at least one of the following: linear multiplication of feature vectors and linear addition of feature vectors.

[0113] It should be noted that the specific feature fusion method can be determined according to actual needs and is not limited in this application.

[0114] In some embodiments of the present application, since the resolutions of the feature maps corresponding to the j-1th detail feature vector and the jth detail feature vector may be different, the first cutout model can first upsample the feature map with a smaller resolution so that the resolutions of the feature maps received by the edge attention module are the same, so that the edge attention module can perform related processing of feature fusion.

[0115] In some embodiments of the present application, the first cutout model may first preprocess the feature map corresponding to the j-th detail feature vector to output two mask images.

[0116] For example, assume that j=2. The first edge attention module in the first cutout model, namely edge attention module 1, can first be based on the second detail feature vector output by the second deconvolution module. The feature map of the second detail feature vector can be Figure 2 For an image similar to the image shown in FIG. 1 , the image regions between 0 and 1 in the feature map of the second detail feature vector are set to white, and the other regions are set to black to obtain an image such as Figure 5A The mask image shown is mask1. Then, the first cutout model can perform color inversion processing based on mask1 to obtain Figure 5BThe mask image shown is mask2. Therefore, the input of edge attention module 1 is: the first detail feature vector output by deconvolution module 1, such as TransFeature1; the second detail feature vector output by deconvolution module 2, such as TransFeature2; mask1 and mask2.

[0117] Furthermore, the edge attention module 1 can perform linear multiplication based on TransFeature1 and mask2 to retain the image portion with higher certainty in the central area of ​​the feature map, and can perform linear multiplication based on TransFeature2 and mask1 to obtain the image portion of the edge area of ​​the image element in the accurate feature map. Then, the edge attention module 1 can linearly add the results of the above linear multiplication, that is, EdgeAtt1 = TransFeature1*mask2 + TransFeature2*mask1, and combine the central area and the edge area of ​​the image element in the feature map, so that the output first edge feature vector EdgeAtt1 can have a certain improvement in the degree of refinement of the cutout at the edge position of the image element.

[0118] Step A2: The electronic device performs upsampling processing on the Njth feature vector, the jth detail feature vector and the j / 2th edge feature vector through the j+1th deconvolution module in the first cutout model, and outputs the j+1th detail feature vector.

[0119] In some embodiments of the present application, the j+1th deconvolution module in the first cutout model can be Perform matrix calculation to upsample the received feature vector to obtain the j+1th detail feature vector TransFeature j+1 .in, It can be the weight value corresponding to the j+1th deconvolution module; x can be the Njth feature vector, the jth detail feature vector and the j / 2th edge feature vector; It can be the bias value corresponding to the j-th deconvolution module.

[0120] For example, in combination Figure 4A ,like Figure 6 As shown, the electronic device can first perform linear addition of TransFeature2 output by deconvolution module 2, Feature2 output by convolution module 2, and EdgeAtt1 output by attention edge module 1 through skip2, that is, skip2 = TransFeature2 + Feature2 + EdgeAtt1, as the input of the third deconvolution module, that is, deconvolution module 3. Therefore, deconvolution module 3 can be Perform matrix calculation to upsample the feature vector corresponding to the received skip2 to obtain the third detail feature vector TransFeature3. It can be the weight value corresponding to the deconvolution module 3; It can be the offset value corresponding to the deconvolution module 3. In addition, the third detail feature vector TransFeature3 is a feature map with a resolution of 256*256, and the feature map is obtained by combining images of 16 channels.

[0121] It should be noted that, during the execution of the above step 202a2, step A1, and step A2, the electronic device sequentially executes the processing corresponding to the modules based on the connection order of the modules in the first cutout model, and at the same time, all the deconvolution modules at even positions in the N deconvolution models execute the content of the above step 202a2, and all the deconvolution modules at odd positions execute the content of the above steps A1 and A2. Then, the electronic device can determine the detail feature vector output by the last deconvolution module as the detail feature map.

[0122] Exemplarily, in the case where the first cutout model includes 6 deconvolution modules and 3 edge attention modules, the electronic device may, after executing step 202a1, execute step 202a2 through the deconvolution module 2, and then execute step A1 through the edge attention module 1, and execute step A2 through the deconvolution module 3; further, execute step 202a2 through the deconvolution module 4, and then execute step A1 through the edge attention module 2, and execute step A2 through the deconvolution module 5; further, execute step 202a2 through the deconvolution module 6, and then execute step A1 through the edge attention module 3.

[0123] For example, in combination Figure 4B and Figure 6 ,like Figure 7 As shown, the electronic device can linearly add TransFeature3 output by deconvolution module 3 and Feature1 output by convolution module 1 through skip3, that is, skip3 = TransFeature3 + Feature1. Therefore, the fourth deconvolution module in the first cutout model, that is, deconvolution module 4, can receive the image data transmitted by skip3. Therefore, deconvolution module 4 can perform linear addition on the output of deconvolution module 3 and Feature1 output by convolution module 1 through skip3, that is, skip3 = TransFeature3 + Feature1. Perform matrix calculation to upsample the received feature vector corresponding to skip3 to obtain the fourth detail feature vector TransFeature4. It can be the weight value corresponding to the deconvolution module 4; It can be the offset value corresponding to the deconvolution module 4. Moreover, the fourth detail feature vector TransFeature4 is a feature map with a resolution of 512*512, and the feature map is obtained by combining images of 16 channels. Therefore, the electronic device can determine the fourth detail feature vector TransFeature4 as a detail feature map.

[0124] In some embodiments of the present application, in combination with the above steps 201a to 201c, the above step 202 of "processing the first feature map to obtain an edge feature map" can be specifically implemented through the following steps 202b1 and 202b2.

[0125] Step 202b1: The electronic device performs feature fusion on the j-1th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first cutout model to obtain the j / 2th edge feature vector.

[0126] In an embodiment of the present application, the above-mentioned j-1th detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, and the jth detail feature vector is the detail feature vector output by the jth deconvolution module in the first cutout model.

[0127] Among them, j∈[2,N], and j is an even number.

[0128] It should be noted that, for the specific process of the electronic device obtaining the j / 2th edge feature vector in step 202b1, reference may be made to the relevant records in the above steps A1 and A2, and the present application will not repeat them here.

[0129] Step 202b2: The electronic device determines the edge feature vector corresponding to the last edge attention module in the first cutout model as an edge feature map.

[0130] In some embodiments of the present application, when N is an even number, the last edge attention module may be the N / 2th edge attention module; when N is an odd number, the last edge attention module may be the (N-1) / 2th edge attention module.

[0131] For example, in combination Figure 7 As shown, the electronic device can use the second edge attention module in the first cutout model, that is, edge attention module 2, to set the image area between 0 and 1 in the feature map of TransFeature3 to white and other areas to black based on TransFeature3 output by deconvolution module 3, so as to obtain an image similar to Figure 5AThe mask image shown is mask3. Then, the first cutout model can perform color inversion processing based on mask3 to obtain a color similar to Figure 5B The mask image shown is mask4. Therefore, the input of edge attention module 2 is: TransFeature3 output by deconvolution module 3, TransFeature4 output by deconvolution module 4, mask3 and mask4. Edge attention module 2 can perform linear multiplication based on TransFeature3 and mask4 to retain the image part with higher certainty in the central area of ​​the feature map, and can perform linear multiplication based on TransFeature4 and mask3 to obtain the image part of the edge area of ​​the image element in the accurate feature map; then, edge attention module 2 can linearly add the results of the above linear multiplication based on EdgeAtt2=TransFeature3*mask4+TransFeature4*mask3 to combine the central area and edge area of ​​the image element in the feature map, so that the output second edge feature vector EdgeAtt2 can have a certain improvement in the degree of refinement of the cutout at the edge position of the image element. At the same time, the electronic device can determine the edge feature vector corresponding to the last edge attention module, that is, EdgeAtt2 corresponding to edge attention module 2 as the edge feature map.

[0132] In this way, the electronic device can improve the segmentation accuracy of the detail position and edge position of the image elements to be cut out by the first cutout model through the deconvolution module and the edge attention module, so that a cutout image with a higher degree of accuracy can be directly obtained in the future without the need for the user to manually draw the edge position of the image element, thereby reducing the steps that the user needs to perform manually. Furthermore, since feature fusion processing is performed on multiple feature maps during the operation of the deconvolution module and the edge attention module, the details of each layer of feature output in the calculation process of the first cutout model can be enriched, thereby further improving the degree of refinement.

[0133] Step 203: The electronic device outputs a first cutout image through a first cutout model based on at least one of the detail feature map and the edge feature map, and the first feature map.

[0134] In some embodiments of the present application, the electronic device may include an output convolution layer.

[0135] In some embodiments of the present application, the electronic device may perform feature fusion processing and upsampling processing based on the first feature map, the detail feature map and the edge feature map through the output convolution layer to output a first cutout image.

[0136] For example, in combination Figure 7, the electronic device can first use skip4 to linearly add the detail feature map TransFeature4 output by the deconvolution module 4, Feature0 output by the input convolution layer, that is, the feature map obtained based on the first image and the second image, and the edge feature map E d g2e output by the attention edge module 2, that is, skip4 = TransFeature4 + Feature0 + EdgeAtt2, as the input of the output convolution layer. Therefore, the output convolution layer can be obtained by Perform matrix calculation to upsample the received feature vector corresponding to skip4 to obtain feature vector TransFeature0, which is the first cutout image. It can be the weight value corresponding to the output convolution layer; It can be the offset value corresponding to the output convolution layer. In addition, the resolution of the first cutout image is 512*512, and the first cutout image is obtained by combining images of 4 channels.

[0137] In some embodiments of the present application, the electronic device may perform feature fusion processing and upsampling processing based on the edge feature map and the first feature map through the output convolution layer to output the corresponding first cutout image.

[0138] For example, in combination Figure 7 The electronic device can use skip4 to linearly add the edge feature map EdgeAtt2 output by the attention edge module 2 and Feature0 output by the input convolution layer, that is, the feature map obtained based on the first image and the second image, that is, skip4=Feature0+EdgeAtt2, as the input of the output convolution layer, that is, the above-mentioned first cutout image.

[0139] In some embodiments of the present application, the electronic device may perform feature fusion processing and upsampling processing based on the detail feature map and the first feature map through the output convolution layer to output the corresponding first cutout image.

[0140] For example, Figure 4B As shown, the electronic device can use skip4 to linearly add the detail feature map TransFeature4 output by the deconvolution module 4 and Feature0 output by the input convolution layer, that is, the feature map obtained based on the first image and the second image, that is, skip4=TransFeature4+Feature0, as the input of the output convolution layer, that is, the above-mentioned first cutout image.

[0141] In the image processing method provided in the embodiment of the present application, since the second image is an image obtained based on the image elements that need to be cut out in the first image, the second image includes the image elements that need to be cut out. Therefore, the electronic device can perform processing based on the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. That is, the first cutout image output by the first cutout model can accurately retain at least one of the detail image content and the edge image content in the image elements that the user needs to cut out in the first image based on the second image, so that the first cutout image can meet the user's demand for the fineness of the cutout image, thereby improving the image quality of the cutout image obtained by the electronic device.

[0142] In some embodiments of the present application, Figure 1 ,like Figure 8 As shown, before the above step 201, the image processing method provided in the embodiment of the present application also includes the following steps 301 to 303.

[0143] Step 301: The electronic device performs a cutout process on a first image element in a first image to obtain a second cutout image.

[0144] In the embodiment of the present application, the second cutout image may include the first image element.

[0145] In some embodiments of the present application, the first image element may be an image element in the first image that the user needs to cut out.

[0146] In some embodiments of the present application, the above-mentioned cutout processing may be an “instant cutout” method.

[0147] It should be noted that the algorithm mainly involved in the above-mentioned "instant cutout" method is the terminal-side salient object detection (SOD) algorithm, which refers to a method of segmenting the main image elements in the image to be cutout to extract the approximate segmentation results of the image elements selected by the user.

[0148] In some embodiments of the present application, when an electronic device displays a first image that needs to be cut out, the electronic device can receive a second input from a user regarding the first image to determine a first image element based on an image area corresponding to the second input, thereby performing cutout processing on the first image element.

[0149] In some embodiments of the present application, the second input may include any one of the following: click input, slide input, long press input, or other feasible input, which is not limited in the embodiments of the present application.

[0150] In some embodiments of the present application, the above-mentioned click input can be any number of click inputs.

[0151] In some embodiments of the present application, the above-mentioned sliding operation can be a sliding operation in any direction, such as sliding up, sliding down, sliding to the left or sliding to the right, etc., which is not limited in the embodiments of the present application.

[0152] In some embodiments of the present application, the electronic device may determine the image area corresponding to the second input according to the input parameters of the second input, thereby determining the first image element.

[0153] In some embodiments of the present application, the above-mentioned input parameters may include but are not limited to at least one of the following: input position, input trajectory.

[0154] For example, the electronic device is a mobile phone. Fig. 9A As shown, the mobile phone can display an image editing interface 10 to display an image 1 that the user wants to edit. In the image 1, a man holding a beverage cup takes a photo with a building in the background. If the user wants to adjust the position of the man in the image 1, as shown in FIG. Fig. 9B As shown, the user can long press the "man" in image 1 to trigger the mobile phone to determine that the user wants to obtain the image of the "man" in image 1 according to the position pressed by the user, and then cut out the image area of ​​the "man" selected by the user through the "instant cutout" method to obtain the image of the "man". Fig. 9C The gray area shown is cut out to obtain a cutout image corresponding to "man", that is, the second cutout image.

[0155] Step 302: The electronic device segments the first image based on image elements in the first image to obtain at least two segmented images.

[0156] In some embodiments of the present application, the electronic device may determine the image elements included in the first image based on the types of the image elements.

[0157] In some embodiments of the present application, the types of the above-mentioned image elements may include, but are not limited to, any of the following: people, animals, objects, buildings, and natural environments.

[0158] In some embodiments of the present application, when an image element includes multiple characters, the electronic device can segment the images corresponding to the multiple characters according to information such as gender, age group, character actions, and character clothing corresponding to the multiple characters.

[0159] In some embodiments of the present application, when an image element includes an animal, the electronic device may segment the image corresponding to the animal according to information such as the type, color, and posture of the animal.

[0160] In some embodiments of the present application, when an image element includes an object, the electronic device can segment the image corresponding to the object according to information such as the shape, color, and purpose of the object.

[0161] In some embodiments of the present application, the electronic device may classify the image elements in the first image by using an end-to-end object detector.

[0162] Exemplarily, the end-to-end object detector may be a dino classification method, or may be a classification method improved on the basis of the dino classification method. For example, the dino model is fine-tuned so that the fine-tuned classification model can be more suitable for the classification of image elements of images stored in the album.

[0163] In some embodiments of the present application, the electrons can still be segmented by a Segment Anything Model (SAM) model based on the type of the image elements in the first image.

[0164] In some embodiments of the present application, the at least two segmented images may include: a segmented image corresponding to at least one image element included in the first image, and a segmented image corresponding to the image background of the first image.

[0165] In some embodiments of the present application, after the electronic device obtains at least two segmented images, the electronic device may add a label to each segmented image including an image element to mark image information corresponding to each image element.

[0166] In some embodiments of the present application, the above-mentioned image information may include but is not limited to at least one of the following: type, location.

[0167] For example, in combination Fig. 9A, the mobile phone can determine that image 1 contains three image elements based on the Dino method, such as a "man", a beverage cup and a building in the background. Then, the mobile phone can segment the three image elements based on the SAM model to obtain four segmented images, such as segmented image 1 to segmented image 4. Among them, segmented image 1 can be a segmented image including "man", segmented image 2 can be a segmented image including a beverage cup, segmented image 3 can be a segmented image including a building, and segmented image 4 can be a segmented image corresponding to the remaining images except the three image elements in image 1. Then, the mobile phone can add labels for indicating the type of image elements to the segmented images including image elements, that is, segmented images 1 to segmented images 3. For example, the mobile phone can add a "person" label to segmented image 1 to indicate that the image element included in segmented image 1 is a person, or the mobile phone can add specific labels such as "young" and "man" to segmented image 1 for describing person information.

[0168] Step 303: The electronic device determines a second image based on the area relationship between the first segmented image and the second cutout image.

[0169] In the embodiment of the present application, the first segmented image may be a segmented image including the first image element among the at least two segmented images.

[0170] In some embodiments of the present application, the method for the electronic device to determine the first segmented image may include but is not limited to any of the following:

[0171] The electronic device may determine the first image element that the user needs to cut out according to the input position of the second input, and then determine the first segmented image according to the image elements included in each of the at least two segmented images;

[0172] The electronic device may determine the first segmented image according to the input position of the second input and the original display positions corresponding to the at least two segmented images;

[0173] The electronic device can also determine the maximum external frame based on the external frame corresponding to the first image element in the second cutout image and the external frames corresponding to the separated images containing the first image element in the at least two segmented images, so as to perform image segmentation on the image elements contained in the maximum external frame and determine the first segmented image.

[0174] In some embodiments of the present application, the electronic device may indicate the display position of the circumscribed frame in the image by means of the coordinates of the upper left vertex and the lower right vertex in the circumscribed frame.

[0175] In some embodiments of the present application, the electronic device may determine the maximum external frame based on the following formula group (1).

[0176]

[0177] Among them, (new_x1, new_y1) can be the vertex of the upper left corner of the above-mentioned maximum circumscribed frame, and (new_x2, new_y2) can be the vertex of the lower right corner of the above-mentioned maximum circumscribed frame; (sod_x1, sod_y1) can be the vertex of the upper left corner of the circumscribed frame of the first image element in the above-mentioned second cutout image, and (sod_x2, sod_y2) can be the vertex of the lower right corner of the first image element in the above-mentioned second cutout image; (label_x1, label_y1) can be the vertex of the upper left corner of the separation image corresponding to the first image element, and (label_x2, label_y2) can be the vertex of the lower right corner of the separation image corresponding to the first image element; img_width can be the width of the first image, and img_height can be the height of the first image.

[0178] In some embodiments of the present application, in the above formula group (1), the electronic device can determine the minimum horizontal value and the minimum vertical value based on the coordinates of the vertex of the upper left corner of the second cutout image and the coordinates of the vertex of the upper left corner of the separated image corresponding to the first image element, so as to combine them into the coordinates of the vertex of the upper left corner of the maximum circumscribed frame, wherein, since the coordinates of the vertex of the upper left corner of the maximum circumscribed frame are located inside the first image, the minimum value of the coordinates of the vertex of the upper left corner of the maximum circumscribed frame is 0, that is, it is located in the upper left corner of the first image.

[0179] In some embodiments of the present application, in the above formula group (1), the electronic device can determine the maximum horizontal value and the maximum vertical value based on the coordinates of the vertex of the lower right corner of the second cutout image and the coordinates between the vertex of the lower right corner of the separated image corresponding to the first image element, so as to combine them into the coordinates of the vertex of the lower right corner of the maximum circumscribed frame, wherein, since the coordinates of the vertex of the lower right corner of the maximum circumscribed frame are located inside the first image, the horizontal coordinate of the vertex of the lower right corner of the maximum circumscribed frame is smaller than the width of the first image, and the vertical coordinate is smaller than the height of the first image.

[0180] For example, assume that the width of the first image is 12 and the height is 15. Fig. 9C , the second cutout image can be Fig. 10AFor the cutout image 11 shown, the mobile phone can establish a coordinate system based on the upper left corner of the cutout image 11 as the coordinate origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. Then, the mobile phone can use the coordinates of the vertex A at the upper left corner of the external frame 12 (sod_x1, sod_y1), such as (5, 7); and the coordinates of the vertex B at the lower right corner (sod_x2, sod_y2), such as (10, 12) to indicate the display position of the external frame 12. The separated image corresponding to the first image element can be as follows: Fig. 10B For the segmented image 13 shown, the mobile phone can establish a coordinate system based on the upper left corner of the segmented image 13 as the coordinate origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. Then, the mobile phone can use the coordinates (label_x1, label_y1) of the vertex C at the upper left corner of the external frame 14, such as (5, 6) and the coordinates (label_x2, label_y2) of the vertex D at the lower right corner, such as (10, 12) to indicate the display position of the external frame 14. Thus, the mobile phone can perform the following calculations:

[0181] new_x1=max(0,min(5,5))=5

[0182] new_y1=max(0,min(7,6))=6

[0183] new_x2=min(12,max(10,10))=10

[0184] new_y2=min(15,max(12,12))=12

[0185] The coordinates of the upper left corner vertex of the maximum circumscribed frame are determined to be (5, 6), and the coordinates of the lower right corner vertex are determined to be (10, 12), which are also the vertex coordinates of the circumscribed frame 14. Then, the mobile phone can determine the circumscribed frame 14 as the maximum circumscribed frame, and perform image segmentation based on the image elements contained in the maximum circumscribed frame to determine the first segmented image.

[0186] In some embodiments of the present application, when the multiple relationship between the first segmented image and the second cutout image is within a preset range, the electronic device may determine the first segmented image as the second image; when the multiple relationship between the first segmented image and the second cutout image is not within the above-mentioned preset range, the electronic device may determine the second cutout image as the second image.

[0187] Exemplarily, it is assumed that the maximum connected domain area of ​​the second cutout image is area1, and the maximum connected domain area of ​​the first segmented image is area2.

[0188]

[0189] That is, when the area of ​​the first segmented image is greater than 1.75 times the area of ​​the second cutout image, the electronic device can determine the second cutout image as the second image; when the area of ​​the first segmented image is less than 0.5 times the area of ​​the second cutout image, the electronic device can determine the second cutout image as the second image; when the area of ​​the first segmented image is 0.5 to 1.75 times the area of ​​the second cutout image, the electronic device can determine the first segmented image as the second image.

[0190] It should be noted that the above 0.5 times and 1.75 times are only used for illustration. The specific numerical setting can be determined according to the needs of users in actual applications, and this application does not limit it here.

[0191] In the embodiment of the present application, the electronic device can obtain multiple cutout results through multiple different cutout methods, so as to avoid the situation that the cutout result stored may contain the image part that the user does not need, or does not completely contain the image part that the user needs, in the process of directly obtaining the cutout result through one cutout method. At the same time, the electronic device can determine the second image to be used subsequently through the ratio of the area of ​​the first segmented image to the second cutout image, so as to avoid the situation that the gap between the two different segmentation results is too large, resulting in a large deviation between the cutout result and the content required by the user.

[0192] In some embodiments of the present application, Figure 1 ,like Fig.11 As shown, after the above step 203, the image processing method provided in the embodiment of the present application further includes the following steps 401 to 403.

[0193] Step 401: The electronic device displays a first cutout image in a first image.

[0194] Step 402: The electronic device receives a first input of a first cutout image.

[0195] In some embodiments of the present application, the first input may be an input that triggers the electronic device to adjust the display position of the first cutout image in the first image.

[0196] In some embodiments of the present application, the above-mentioned first input may include any of the following: click input, sliding input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0197] In some embodiments of the present application, the above-mentioned click input can be any number of click inputs.

[0198] In some embodiments of the present application, the above-mentioned sliding operation can be a sliding operation in any direction, such as sliding up, sliding down, sliding to the left or sliding to the right, etc., which is not limited in the embodiments of the present application.

[0199] In some embodiments of the present application, the above-mentioned gesture input may include but is not limited to at least one of the following: click gesture, sliding gesture, drag gesture, pressure recognition gesture, long press gesture, area change gesture, double press gesture, double click gesture, specific gesture input or other possible gesture inputs. The specific gesture input form can be determined according to actual needs and is not limited to this in the embodiments of the present application.

[0200] Step 403: In response to the first input, the electronic device adjusts the display position of the first cutout image in the first image, and fills the original position of the first cutout image with an image.

[0201] In some embodiments of the present application, during the process of the electronic device adjusting the display position of the first cutout image, the electronic device may adjust the display size of the first cutout image according to the user input.

[0202] In some embodiments of the present application, after the electronic device adjusts the display position of the first cutout image according to the user's input, if the electronic device does not receive the user's input within a preset time period, the electronic device may start to fill the original position of the first cutout image with an image. For example, the preset time period may be 3 seconds.

[0203] It should be noted that the above-mentioned preset time period can be a default time length set for the electronic device, or can be set according to user needs, and this application does not limit this.

[0204] In some embodiments of the present application, the electronic device may fill the original position of the first cutout image with an image through a diffusion model, wherein the diffusion model (Stable Diffusion) may include an image internal filling (Inpaint) module and an image external filling (Outpaint) module.

[0205] It should be noted that the above diffusion model is a deep learning method based on the diffusion model, which can achieve the conversion from text to image by learning the mapping relationship between a large amount of image and text data. In addition, the image structure and texture consistency can be maintained in the process of generating images based on the diffusion model;

[0206] The working principle of the above Inpaint module can be summarized in two steps: first, the user needs to provide an image to be repaired and draw a mask on the image to indicate the area that needs to be repaired; then, the deep learning algorithm can generate a new image based on the provided image and mask, in which the defective area is repaired or repainted;

[0207] The image processing flow of the Outpaint module is similar to that of the Inpaint module, but the algorithm processing goal of the Outpaint module is to complete the details of the area outside the picture.

[0208] For example, in combination with 9A, if the user wants to adjust the display position of "Man" in Image 1 and reduce the display size of "Man", the user can long press the display area of ​​"Man" in Image 1 to trigger the mobile phone to obtain the corresponding first cutout image based on "Man" through the first cutout model. Then, the user can use two-finger pinch input to trigger the mobile phone to reduce the display size of the first cutout image according to the user's needs. Then, the user can use drag input to trigger the mobile phone to adjust the display position of the first cutout image. Then, the mobile phone can fill the original position of the first cutout image with an image to display as follows: Fig.12 Image 15 shown.

[0209] For example, Fig.13A As shown, assuming that the first image is an image 16 containing a teddy bear and a building, and in image 16 the teddy bear is located at the right edge of the image, if the user wants to adjust the display position of the teddy bear to the center of image 16, the user can long press the display area of ​​the teddy bear in image 16 to trigger the mobile phone to obtain a corresponding cutout image based on the teddy bear through the first cutout model, and then the user can trigger the mobile phone to adjust the display position of the cutout image corresponding to the teddy bear by dragging input. Then, the mobile phone can fill the background building at the original position of the cutout image corresponding to the teddy bear, and can complete the teddy bear based on the content in the cutout image of the teddy bear, so as to display the following: Fig. 13B Image 17 shown.

[0210] In the embodiment of the present application, the electronic device can adjust the display parameters of the first cutout image in the first image, and can fill the original display position of the first cutout image with an image, so that the user can recompose the first image so that the adjusted first image meets the actual needs of the user. This can improve the convenience of image processing by the electronic device.

[0211] In some embodiments of the present application, in combination with the above steps 401 to 403, the image processing method provided in the embodiments of the present application further includes the following step 404.

[0212] Step 404: The electronic device adjusts the display parameters of the first matte image based on the display parameters of the adjacent regions of the first matte image after the position adjustment.

[0213] In some embodiments of the present application, the above display parameters may include, but are not limited to, any one of the following: image brightness, image saturation, and image color level.

[0214] In some embodiments of the present application, before adjusting the display parameters of the first matte image, the electronic device may perform image processing on the edge region of the first matte image to weaken the sense of jerk at the edge of the first matte image.

[0215] In some embodiments of the present application, the electronic device may calculate the central brightness of the image based on the main region of the first image and the original image background region corresponding to the first matte image after the display position adjustment, so as to adjust the color level curve of the first matte image, so that the brightness of the first matte image is similar to that of the adjacent regions.

[0216] In some embodiments of the present application, the electronic device may represent the first image through the LAB color model, and then extract the L channel in the LAB color model, that is, the brightness channel, so as to define the average value of the brightness values of all pixels in the first image through the average brightness mean_light, so that different color level outputs can be adjusted through different average brightness values, so that the main color of the first matte image is adapted to the color of the adjacent regions after the position adjustment. Among them,

[0217] Among them, the Lab color model consists of three elements. L represents brightness, and A and B are two color channels. Among them, the colors included in A are from dark green to gray to bright pink, that is, from low brightness values to medium brightness values to high brightness values; B is from bright blue to gray to yellow, that is, from low brightness values to medium brightness values to high brightness values.

[0218] In some embodiments of the present application, when the difference between the average brightness of the first matte image and the average brightness of the adjacent regions of the first matte image after the position adjustment is greater than a preset threshold, the electronic device may adjust the display parameters of the first matte image. For example, the above preset threshold may be 50.

[0219] Exemplarily, assume that the average brightness of the first matte image is mean_light, and the average brightness of the adjacent regions of the first matte image after the position adjustment is mean_light2. Then, when mean_light < mean_light2 - 50 or mean_light > mean_light2 + 50, the electronic device may adjust the color level range of the first matte image.

[0220] It should be noted that the setting of the above preset threshold can be determined according to actual needs and is not limited in this application.

[0221] In an embodiment of the present application, the electronic device can adjust the display parameters of the first cutout image so that the moved first cutout image can match the display parameters of the moved adjacent area, thereby improving the display effect of the adjusted first image.

[0222] In some embodiments of the present application, the image processing method provided in the embodiments of the present application also includes the following steps 501 to 503.

[0223] Step 501: The electronic device inputs a sample image into a second cutout model, performs cutout processing on a second image element in the sample image, and obtains a third cutout image.

[0224] Step 502: The electronic device calculates a loss value based on the third cutout image and the target cutout image.

[0225] In the embodiment of the present application, the above loss value represents the detail feature difference and edge feature difference between the third cutout image and the target cutout image.

[0226] Step 503: The electronic device adjusts the model parameters in the second cutout model based on the loss value to obtain the first cutout model.

[0227] In some embodiments of the present application, the sample images may be images with a sufficiently large number of samples, for example, 5,000 sample images.

[0228] It should be noted that the specific number of sample images can be determined according to the actual needs of the model training process, and this application does not limit it here.

[0229] In some embodiments of the present application, the structure of the second cutout model is substantially the same as that of the first cutout model. For details, please refer to the description of the above embodiments, and the embodiments of the present application will not be repeated here.

[0230] In some embodiments of the present application, the target cutout image may be a cutout image obtained by manual drawing, which has a high degree of refinement.

[0231] It should be noted that, the smaller the difference between the third cutout image and the target cutout image, the closer the display effect of the third cutout image is to the cutout image required by the user, and the smaller the loss value is.

[0232] In some embodiments of the present application, the electronic device may use a back propagation algorithm based on the loss value to iteratively update the weight values ​​and bias values ​​in the second cutout model until convergence, so as to train the first cutout model.

[0233] Exemplarily, assume that the second cutout model includes 1 input convolution layer, 4 convolution modules, 4 deconvolution modules, 1 output convolution layer and 2 edge attention modules. The second cutout model can downsample the input image 5 times through the 1 input convolution layer and 4 convolution modules, and then input the first feature map output by the fourth convolution module into the deconvolution module 1 among the 4 deconvolution modules for upsampling to obtain detail feature vector 1; then, the electronic device can linearly add the detail feature vector 1 output by the deconvolution module 1 and the feature vector 3 output by the convolution module 3 through the jump connection 1 as the input of the deconvolution module 2, perform upsampling to obtain detail feature vector 2; then, the electronic device can input the detail feature vector 1 and the detail feature vector 2 into the edge attention module 1 for feature fusion, and output the edge feature vector 1; then, the jump connection 2 can be based on the detail feature vector 2 output by the deconvolution module 2 and the feature vector 2 output by the convolution module 2. The edge feature vector 1 output by the edge attention module 1 is linearly added to the edge feature vector 1, and an upsampling process is performed through the deconvolution module 3 to obtain a detail feature vector 3; then, the electronic device can linearly add the detail feature vector 3 output by the deconvolution module 3 and the feature vector 1 output by the convolution module 1 through the jump connection 3, and use them as the input of the deconvolution module 4, and perform an upsampling process to obtain a detail feature vector 4; then, the electronic device can input the detail feature vector 3 and the detail feature vector 4 into the edge attention module 2 for feature fusion, and output the edge feature vector 2; then, the jump connection 4 can linearly add the detail feature vector 4 output by the deconvolution module 4, the feature vector output by the input convolution layer, and the edge feature vector 2 output by the edge attention module 2, and perform an upsampling process through the output convolution layer to output a third cutout image.

[0234] Furthermore, the second detection model can calculate the loss value based on the third cutout image and the target cutout image, and then based on the loss value, the weight value and bias value in the second cutout model are iteratively updated using the back propagation algorithm until convergence, so as to train the first cutout model.

[0235] For various scenarios to which the embodiments of the present application can be applied, combined with the various implementation schemes of the embodiments of the present application described above, specific examples are given below to illustrate the implementation process of the embodiments of the present application in various scenarios.

[0236] Example 1: Perform secondary composition on the person in the image.

[0237] If the user wants to recompose an image, such as adjusting the position of person 1 in image 1 relative to the background building, the user can trigger the electronic device to obtain the cutout image corresponding to image 1 based on the existing "instant cutout", and input image 1 and the cutout image corresponding to image 1 into the first cutout model, so that the first cutout model outputs the first cutout image corresponding to person 1 with detail features and edge features based on the input image, such as cutout image 1. Then, the electronic device can adjust the display position of cutout image 1 according to the user's input of cutout image 1, and can fill the original display position of cutout image 1 according to the background building, and adjust the parameters such as the brightness of the moved cutout image 1, so that the user can obtain the required image 1, and realize the secondary composition of image 1.

[0238] Example 2: Perform secondary composition on objects in the image.

[0239] If the user wants to recompose an image, such as adjusting an object in image 2, such as the display position and display size of object 2, the user can trigger the electronic device to obtain the cutout image corresponding to image 2 based on the existing "instant cutout", and input image 2 and the cutout image corresponding to image 2 into the first cutout model, so that the first cutout model outputs the first cutout image corresponding to object 2 with detail features and edge features based on the input image, such as cutout image 2. Then, the electronic device can adjust the display position and display size of cutout image 2 according to the user's input of cutout image 2, and can fill the original display position of cutout image 2 according to other image elements in image 2, and adjust parameters such as brightness of cutout image 2 after movement, so that the user can obtain the required image 2, and realize the secondary composition of image 2.

[0240] The present application embodiment provides a method for training a cutout model. Fig.14 FIG. 1 is a flowchart of a method for training a cutout model provided in an embodiment of the present application, which method can be applied to electronic devices. Fig.14 As shown, the cutout model training method provided in the embodiment of the present application may include the following steps 601 to 603.

[0241] Step 601: The electronic device inputs a sample image into a second cutout model, performs a cutout process on a second image element in the sample image, and obtains a third cutout image.

[0242] Step 602: The electronic device calculates a loss value based on the third cutout image and the target cutout image.

[0243] In the embodiment of the present application, the above loss value represents the detail feature difference and edge feature difference between the third cutout image and the target cutout image.

[0244] Step 603: The electronic device adjusts the model parameters in the second cutout model based on the loss value to obtain the first cutout model.

[0245] It should be noted that the cutout model training method of the above steps 601 to 603 can be specifically referred to the description of steps 501 to 503 in the above embodiment, and this application will not elaborate on it here.

[0246] In the cutout model training method provided in the embodiment of the present application, since the first cutout model can be trained based on the second cutout model in combination with the detail features and edge features of the cutout image, in actual use, the electronic device can input the second image obtained based on the image elements to be cutout and the original first image into the first cutout model, so that the user can obtain the first cutout image having at least one of the detail features and the edge features, so that the first cutout image can meet the user's demand for the fineness of the cutout image, thereby improving the image quality of the cutout image obtained by the image processing device.

[0247] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0248] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0249] Fig.15 A possible structural diagram of an image processing device involved in an embodiment of the present application is shown. Fig.15 As shown, the image processing device 70 may include: an acquisition module 71, a processing module 72 and an output module 73;

[0250] The acquisition module 71 is used to input the first image and the second image into the first cutout model, and acquire the first feature map based on the first image and the second image through the first cutout model, where the second image is an image obtained based on the image elements in the first image;

[0251] A processing module 72, configured to process the first feature map obtained by the obtaining module 71 by using a first cutout model to obtain at least one of a detail feature map and an edge feature map;

[0252] An output module 73, configured to output a first cutout image through a first cutout model based on at least one of the detail feature map and the edge feature map acquired by the acquisition module and the first feature map;

[0253] Among them, the first cutout model is trained based on the detail features and edge features of the image.

[0254] In one possible implementation, the image processing device 70 provided in the embodiment of the present application also includes: a determination module; the above-mentioned processing module 72 is also used to perform cutout processing on the first image element in the first image before the acquisition module 71 inputs the first image and the second image into the first cutout model to obtain a second cutout image, and the second cutout image includes the first image element; and based on the image elements in the first image, the first image is segmented to obtain at least two segmented images; the above-mentioned determination module is used to determine the second image based on the area relationship between the first segmented image and the second cutout image obtained by the processing module 72; wherein the first segmented image is a segmented image including the first image element in at least two segmented images.

[0255] In one possible implementation, the acquisition module 71 is specifically used to: downsample the first image and the second image through the first convolution module in the first cutout model, and output the first feature vector; and downsample the i-1th feature vector through the i-th convolution module in the first cutout model, and output the i-th feature vector, where i∈[2,N], i is an integer, and N is the number of convolution modules; and determine the N-th feature vector as the first feature map.

[0256] In one possible implementation, the processing module 72 is specifically used to: upsample the first feature map through the first deconvolution module in the first cutout model to obtain the first detail feature vector; and upsample the Njth feature vector and the j-1th detail feature vector through the jth deconvolution module in the first cutout model to output the jth detail feature vector, where j∈[2,N]; and determine the Nth detail feature vector as the detail feature map.

[0257] In one possible implementation, the processing module 72 is further used to: after upsampling the Njth feature vector and the j-1th detail feature vector and outputting the jth detail feature vector, perform feature fusion on the j-1th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first cutout model to obtain the j / 2th edge feature vector, where j is an even number; and perform upsampling on the Njth feature vector, the jth detail feature vector and the j / 2th edge feature vector through the j+1th deconvolution module in the first cutout model to output the j+1th detail feature vector.

[0258] In one possible implementation, the above-mentioned processing module 72 is specifically used to: perform feature fusion on the j-1th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first cutout model to obtain the j / 2th edge feature vector, the j-1th detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, and the jth detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, where j∈[2,N] and j is an even number; and determine the edge feature vector corresponding to the last edge attention module in the first cutout model as the edge feature map.

[0259] In one possible implementation, the image processing device 70 provided in the embodiment of the present application also includes: a display module and a receiving module; the display module is used to display the first cutout image in the first image after outputting the first cutout image based on at least one of the detail feature map and the edge feature map, and the first feature map; the receiving module is used to receive a first input of the first cutout image displayed by the display module; the above-mentioned processing module 72 is also used to adjust the display position of the first cutout image in the first image in response to the first input received by the receiving module, and to perform image filling on the original position of the first cutout image.

[0260] In a possible implementation, the processing module 72 is further configured to adjust display parameters of the first cutout image based on display parameters of adjacent areas of the first cutout image after position adjustment.

[0261] In one possible implementation, the image processing device 70 provided in the embodiment of the present application also includes: a calculation module; the above-mentioned acquisition module 71 is also used to input the sample image into the second cutout model, perform cutout processing on the second image element in the sample image, and obtain a third cutout image; the calculation module is used to calculate the loss value based on the third cutout image and the target cutout image acquired by the acquisition module 71, and the loss value represents the detail feature difference and edge feature difference between the third cutout image and the target cutout image; the above-mentioned processing module 72 is also used to adjust the model parameters in the second cutout model based on the loss value to obtain the first cutout model.

[0262] In the image processing device provided in the embodiment of the present application, since the second image is an image obtained based on the image elements that need to be cut out in the first image, the second image includes the image elements that need to be cut out. Therefore, the image processing device can perform processing based on the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. That is, the first cutout image output by the first cutout model can accurately retain at least one of the detail image content and the edge image content in the image elements that the user needs to cut out in the first image based on the second image, so that the first cutout image can meet the user's demand for the fineness of the cutout image, thereby improving the image quality of the cutout image obtained by the image processing device.

[0263] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0264] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0265] The image processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0266] Alternatively, if Fig.16 As shown, an embodiment of the present application further provides an electronic device 90, including a processor 91 and a memory 92, wherein the memory 92 stores programs or instructions that can be executed on the above-mentioned processor 91, and when the program or instructions are executed by the processor 91, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0267] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0268] Fig.17 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0269] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0270] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.17 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0271] The processor 110 is used to input the first image and the second image into the first cutout model, and obtain a first feature map based on the first image and the second image through the first cutout model, where the second image is an image obtained based on image elements in the first image; and process the first feature map through the first cutout model to obtain at least one of a detail feature map and an edge feature map; and output a first cutout image based on at least one of the detail feature map and the edge feature map and the first feature map through the first cutout model; wherein the first cutout model is trained based on the detail features and edge features of the image.

[0272] Optionally, the processor 110 is further used to, before inputting the first image and the second image into the first cutout model, perform cutout processing on the first image element in the first image to obtain a second cutout image, the second cutout image including the first image element; and segment the first image based on the image elements in the first image to obtain at least two segmented images, the at least two segmented images including: a segmented image corresponding to at least one image element included in the first image and a segmented image corresponding to the image background of the first image; and, based on the area relationship between the first segmented image and the second cutout image, determine the second image; wherein the first segmented image is a segmented image including the first image element in the at least two segmented images.

[0273] Optionally, the processor 110 is specifically used to downsample the first image and the second image through the first convolution module in the first cutout model to output a first feature vector; and downsample the i-1th feature vector through the i-th convolution module in the first cutout model to output the i-th feature vector, where i∈[2,N], i is an integer, and N is the number of convolution modules; and determine the N-th feature vector as the first feature map.

[0274] Optionally, the processor 110 is specifically used to upsample the first feature map through the first deconvolution module in the first cutout model to obtain a first detail feature vector; and upsample the Njth feature vector and the j-1th detail feature vector through the jth deconvolution module in the first cutout model to output the jth detail feature vector, where j∈[2,N]; and determine the Nth detail feature vector as a detail feature map.

[0275] Optionally, the processor 110 is further used to, after upsampling the Njth feature vector and the j-1th detail feature vector and outputting the jth detail feature vector, perform feature fusion on the j-1th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first cutout model to obtain the j / 2th edge feature vector, where j is an even number; and upsample the Njth feature vector, the jth detail feature vector and the j / 2th edge feature vector through the j+1th deconvolution module in the first cutout model to output the j+1th detail feature vector.

[0276] Optionally, the processor 110 is specifically used to: perform feature fusion on the j-1th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first cutout model to obtain the j / 2th edge feature vector, the j-1th detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, and the jth detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, where j∈[2,N] and j is an even number; and determine the edge feature vector corresponding to the last edge attention module in the first cutout model as the edge feature map.

[0277] Optionally, the display unit 106 is used to display the first cutout image in the first image after outputting the first cutout image based on at least one of the detail feature map and the edge feature map, and the first feature map; the user input unit 107 is used to receive a first input to the first cutout image; the display unit 106 is also used to adjust the display position of the first cutout image in the first image in response to the first input, and to perform image filling on the original position of the first cutout image.

[0278] Optionally, the processor 110 is further configured to adjust display parameters of the first cutout image based on display parameters of adjacent areas of the first cutout image after position adjustment.

[0279] Optionally, the processor 110 is also used to input the sample image into the second cutout model, perform cutout processing on the second image element in the sample image, and obtain a third cutout image; and calculate a loss value based on the third cutout image and the target cutout image, the loss value representing the detail feature differences and edge feature differences between the third cutout image and the target cutout image; and based on the loss value, adjust the model parameters in the second cutout model to obtain the first cutout model.

[0280] In the electronic device provided in the embodiment of the present application, since the second image is an image obtained based on the image elements that need to be cut out in the first image, the second image includes the image elements that need to be cut out. Therefore, the electronic device can perform processing based on the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. That is, the first cutout image output by the first cutout model can accurately retain at least one of the detail image content and the edge image content in the image elements that the user needs to cut out in the first image based on the second image, so that the first cutout image can meet the user's demand for the fineness of the cutout image, thereby improving the image quality of the cutout image obtained by the electronic device.

[0281] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0282] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.

[0283] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0284] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0285] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0286] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0287] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0288] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0289] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0290] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0291] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0292] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0293] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Inputting the first image and the second image into a first matting model, and obtaining a first feature map based on the first image and the second image through the first matting model, wherein the second image is an image obtained based on image elements in the first image; Processing the first feature map by using the first cutout model to obtain at least one of a detail feature map and an edge feature map; Outputting a first matting image based on at least one of the detail feature map and the edge feature map and the first feature map by the first matting model; The first cutout model is trained based on the detail features and edge features of the image.

2. The method according to claim 1, characterized in that Before inputting the first image and the second image into the first cutout model, the method further includes: Performing a cutout process on a first image element in the first image to obtain a second cutout image, wherein the second cutout image includes the first image element; Segmenting the first image based on image elements in the first image to obtain at least two segmented images; determining the second image based on an area relationship between the first segmented image and the second cutout image; The first segmented image is a segmented image including the first image element among the at least two segmented images.

3. The method according to claim 1, characterized in that The obtaining a first feature map based on the first image and the second image by using the first cutout model includes: Down-sampling the first image and the second image through a first convolution module in the first cutout model, and outputting a first feature vector; Downsampling the i-1th feature vector through the i-th convolution module in the first cutout model, and outputting the i-th feature vector, where i∈[2,N], i is an integer, and N is the number of convolution modules; The Nth eigenvector is determined as the first feature map.

4. The method according to claim 3, characterized in that The step of processing the first feature map by using the first cutout model to obtain a detail feature map includes: Performing upsampling processing on the first feature map through a first deconvolution module in the first cutout model to obtain a first detail feature vector; Upsampling the Nj-th feature vector and the j-1-th detail feature vector is performed through the j-th deconvolution module in the first cutout model, and the j-th detail feature vector is output, where j∈[2,N]; The Nth detail feature vector is determined as the detail feature map.

5. The method according to claim 4, characterized in that After up-sampling the Nj-th feature vector and the j-1-th detail feature vector to output the j-th detail feature vector, the method further includes: By using the j / 2th edge attention module in the first cutout model, feature fusion is performed on the j-1th detail feature vector and the jth detail feature vector to obtain the j / 2th edge feature vector, where j is an even number; The Njth feature vector, the jth detail feature vector and the j / 2th edge feature vector are upsampled by the j+1th deconvolution module in the first cutout model to output the j+1th detail feature vector.

6. The method according to any one of claims 3 to 5, characterized in that The step of processing the first feature map by using the first cutout model to obtain an edge feature map includes: Performing feature fusion on the j-1th detail feature vector and the j-th detail feature vector through the j / 2th edge attention module in the first cutout model to obtain the j / 2th edge feature vector, wherein the j-1th detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, and the j-th detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, wherein j∈[2,N], and j is an even number; The edge feature vector corresponding to the last edge attention module in the first cutout model is determined as the edge feature map.

7. The method according to claim 1, characterized in that After outputting a first cutout image based on at least one of the detail feature map and the edge feature map and the first feature map, the method further includes: displaying the first cutout image in the first image; Receiving a first input of the first cutout image; In response to the first input, in the first image, a display position of the first cutout image is adjusted, and an image filling is performed on an original position of the first cutout image.

8. The method according to claim 7, characterized in that The method further comprises: Based on the display parameters of the adjacent area of ​​the first cutout image after the position is adjusted, the display parameters of the first cutout image are adjusted.

9. The method according to claim 1, characterized in that: The method further comprises: Inputting the sample image into a second cutout model, performing cutout processing on the second image element in the sample image, and obtaining a third cutout image; Based on the third matting image and the target matting image, calculating a loss value, wherein the loss value represents a detail feature difference and an edge feature difference between the third matting image and the target matting image; Based on the loss value, the model parameters in the second cutout model are adjusted to obtain the first cutout model.

10. An image processing device, characterized in that: The image processing device comprises: an acquisition module, a processing module and an output module; The acquisition module is used to input the first image and the second image into a first matting model, and acquire a first feature map based on the first image and the second image through the first matting model, where the second image is an image obtained based on image elements in the first image; The processing module is used to process the first feature map obtained by the obtaining module through the first cutout model to obtain at least one of a detail feature map and an edge feature map; The output module is configured to output a first cutout image based on at least one of the detail feature map and the edge feature map acquired by the acquisition module and the first feature map through the first cutout model; The first cutout model is trained based on the detail features and edge features of the image.

11. The device according to claim 10, characterized in that The image processing device further includes: a determination module; The processing module is further configured to perform a cutout process on a first image element in the first image to obtain a second cutout image before the acquisition module inputs the first image and the second image into the first cutout model, wherein the second cutout image includes the first image element; and segment the first image based on the image element in the first image to obtain at least two segmented images; The determining module is used to determine the second image based on the area relationship between the first segmented image and the second cutout image obtained by the processing module; The first segmented image is a segmented image including the first image element among the at least two segmented images.

12. The device according to claim 10, characterized in that The acquisition module is specifically used for: Down-sampling the first image and the second image through a first convolution module in the first cutout model, and outputting a first feature vector; Downsampling the i-1th feature vector through the i-th convolution module in the first cutout model, and outputting the i-th feature vector, where i∈[2,N], i is an integer, and N is the number of convolution modules; The Nth eigenvector is determined as the first feature map.

13. The device according to claim 12, characterized in that The processing module is specifically used for: Performing upsampling processing on the first feature map through a first deconvolution module in the first cutout model to obtain a first detail feature vector; Upsampling the Nj-th feature vector and the j-1-th detail feature vector is performed through the j-th deconvolution module in the first cutout model, and the j-th detail feature vector is output, where j∈[2,N]; The Nth detail feature vector is determined as the detail feature map.

14. The device according to claim 13, characterized in that The processing module is further used for: After upsampling the Nj-th feature vector and the j-1-th detail feature vector to output the j-th detail feature vector, the j-1-th detail feature vector and the j-th detail feature vector are subjected to feature fusion by the j / 2-th edge attention module in the first cutout model to obtain the j / 2-th edge feature vector, where j is an even number; The Njth feature vector, the jth detail feature vector and the j / 2th edge feature vector are upsampled by the j+1th deconvolution module in the first cutout model to output the j+1th detail feature vector.

15. The device according to any one of claims 12 to 14, characterized in that The processing module is specifically used for: Performing feature fusion on the j-1th detail feature vector and the j-th detail feature vector through the j / 2th edge attention module in the first cutout model to obtain the j / 2th edge feature vector, wherein the j-1th detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, and the j-th detail feature vector is the detail feature vector output by the j-1th deconvolution module in the first cutout model, wherein j∈[2,N], and j is an even number; The edge feature vector corresponding to the last edge attention module in the first cutout model is determined as the edge feature map.

16. The device according to claim 10, characterized in that The image processing device further includes: a display module and a receiving module; The display module is configured to display the first cutout image in the first image after outputting the first cutout image based on at least one of the detail feature map and the edge feature map and the first feature map; The receiving module is used to receive a first input of the first cutout image displayed by the display module; The processing module is further configured to, in response to the first input received by the receiving module, adjust a display position of the first cutout image in the first image and perform image filling on an original position of the first cutout image.

17. The device according to claim 16, characterized in that The processing module is further configured to adjust display parameters of the first cutout image based on display parameters of adjacent areas of the first cutout image after position adjustment.

18. The device according to claim 10, characterized in that The image processing device further includes: a calculation module; The acquisition module is further used to input the sample image into a second cutout model, perform cutout processing on the second image element in the sample image, and obtain a third cutout image; The calculation module is used to calculate a loss value based on the third cutout image and the target cutout image acquired by the acquisition module, wherein the loss value represents a detail feature difference and an edge feature difference between the third cutout image and the target cutout image; The processing module is further used to adjust the model parameters in the second cutout model based on the loss value to obtain the first cutout model.

19. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 9 are implemented.

20. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 9 are implemented.

21. A computer program product, characterized in that The computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the image processing method according to any one of claims 1 to 9.

Images