Method executed by electronic device, electronic device, storage medium and program product

Abstract

The invention provides a method executed by electronic equipment, the electronic equipment, a storage medium and a program product, and relates to the fields of computer vision, image processing, artificial intelligence and the like. The method comprises the following steps: extracting at least one object center region and at least one object edge region from a first image, and matching the extracted object center region and object edge region to obtain an object segmentation result of the first image. Optionally, the method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method executed by an electronic device, an electronic device, a storage medium, and a program product. Background Art

[0002] With the rapid development of self-media, communication software and other fields, mobile phones have become an important tool for users to record and share their lives, which has also made users put forward higher and higher requirements on the functions of mobile phone cameras. Image segmentation, as a basic task of mobile image processing, is a key support for downstream tasks.

[0003] In the existing technology, the multi-object image segmentation task cannot extract fine edges, and the fine image segmentation task can only focus on one object and cannot achieve the segmentation of multiple objects at the same time. How to achieve fine multi-object image segmentation task has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the embodiments of the present application is to solve the technical problem of how to achieve a sophisticated multi-object image segmentation task.

[0005] According to one aspect of an embodiment of the present application, a method performed by an electronic device is provided, the method comprising:

[0006] extracting at least one object center region and at least one object edge region from the first image;

[0007] The extracted object center area and the object edge area are matched to obtain the object segmentation result of the first image.

[0008] According to another aspect of an embodiment of the present application, an electronic device is provided. The electronic device includes: a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the method provided in the embodiment of the present application.

[0009] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0010] According to another aspect of the embodiments of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method provided in the embodiments of the present application is implemented.

[0011] The method, electronic device, storage medium and program product executed by the electronic device provided in the embodiments of the present application extract at least one object center region and at least one object edge region from the first image, match the extracted object center region and object edge region, and obtain the object segmentation result of the first image. That is, the object in the first image is divided into the object center region and the object edge region, and the fine image segmentation of multiple objects is achieved by extracting and matching the object center region and the object edge region respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in describing the embodiments of the present application.

[0013] Figure 1 A flowchart of a method executed by an electronic device provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of an edge area and a center area of ​​an object provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of a flow chart of a panoramic fine segmentation method provided in an embodiment of the present application;

[0016] Figure 4a A schematic diagram of a traditional convolution calculation provided in an embodiment of the present application;

[0017] Figure 4b A schematic diagram of a dynamic convolution effect provided in an embodiment of the present application;

[0018] Figure 4c A schematic diagram of a dynamic convolution mechanism provided in an embodiment of the present application;

[0019] Figure 4d A schematic diagram of a dynamic convolution calculation provided in an embodiment of the present application;

[0020] Figure 5 A schematic diagram of an edge dilation convolution provided in an embodiment of the present application;

[0021] Figure 6 A schematic diagram of a method for adding semantics to a texture provided in an embodiment of the present application;

[0022] Figure 7 A schematic diagram of an edge feature attention module provided in an embodiment of the present application;

[0023] Figure 8 A schematic diagram of a cross-attention effect provided in an embodiment of the present application;

[0024] Fig. 9A schematic diagram of an edge and center area extraction module provided in an embodiment of the present application;

[0025] Fig.10 A schematic diagram of matching the central area and the edge area provided in an embodiment of the present application;

[0026] Fig.11 A schematic diagram of a fine mask matching module provided in an embodiment of the present application;

[0027] Fig.12 A schematic diagram of a complete panoramic fine segmentation method provided in an embodiment of the present application;

[0028] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present application as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present application. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0030] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present application. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present application is provided for illustration purposes only and not to limit the purpose of the present application as defined in the appended claims and their equivalents.

[0031] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may establish a connection relationship through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0032] The term "include" or "may include" refers to the presence of corresponding disclosed functions, operations, or components that can be used in various embodiments of the present application, rather than limiting the presence of one or more additional functions, operations, or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0033] The term "or" used in various embodiments of the present application includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, and may also be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0034] Unless defined differently, all terms (including technical terms or scientific terms) used in this application have the same meanings understood by those skilled in the art of this application. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally, unless clearly defined in this application.

[0035] At least some functions of the device or electronic device provided in the embodiments of the present application can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0036] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0037] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0038] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0039] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0040] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0041] The method provided in this application may involve one or more technical fields such as speech, language, image, video or data intelligence.

[0042] Optionally, when it comes to the field of images or videos, according to the present application, in a method performed in an electronic device, a method for image segmentation can obtain output data after object area extraction in an image by using image data as input data of an artificial intelligence model. The artificial intelligence model can be obtained by training. Here, "obtained by training" means obtaining a predefined operation rule or artificial intelligence model configured to perform the desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm. The method of the present application may relate to the field of visual understanding of artificial intelligence technology, which is a technology for identifying and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / positioning, or image enhancement.

[0043] Post-processing based on image segmentation algorithm is the mainstream direction of image processing, and the improvement and optimization of image segmentation algorithm is the development trend of image processing. The embodiment of the present application provides an efficient image segmentation method for obtaining fine mask prediction of multiple objects in an image.

[0044] The following describes several optional embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0045] In an embodiment of the present application, a method executed by an electronic device is provided, such as Figure 1 As shown, the method includes:

[0046] Step S101: extracting at least one object center region and at least one object edge region from a first image;

[0047] In the embodiment of the present application, the first image refers to an image including at least one object for which the segmentation task is to be performed. Optionally, the first image may be a stored image, such as an image selected in an album, or the first image may be an image to be stored, such as an image taken in real time by a camera, an image downloaded from the network, an image received from other devices, etc. The embodiment of the present application does not specifically limit the source of the first image.

[0048] For the embodiment of the present application, the type of the object in the first image is not specifically limited. As an example, the object to be segmented may include a semantic object (stuff) with a non-fixed and uncountable shape, such as the sky, grass, etc.; and / or may include an instance object (thing) with a regular and countable shape, such as a person, a pet, etc., but is not limited thereto.

[0049] In an embodiment of the present application, the object in the first image is divided into an object center region (also referred to as an object entity region, or simply referred to as a center region or entity region) and an object edge region (also referred to as an object boundary region, or simply referred to as an edge region or boundary region), and the edge region and the center region of the segmented object are extracted and predicted respectively. Among them, the object edge region focuses on the edge texture of the object and pays attention to the local features obtained from the edge details; while the object center region focuses on the internal region of the object, pays attention to the global information extracted from the panoramic object, and does not pay attention to the edge detail information. Optionally, each object edge region can be an annular region.

[0050] Step S102: matching the extracted object center region and the object edge region to obtain an object segmentation result of the first image.

[0051] In the embodiment of the present application, the object center area and the object edge area of ​​the segmented object in the first image are matched to obtain a detailed segmentation result for each object.

[0052] As an example, Figure 2As shown, by extracting, predicting and matching the edge area and the center area of ​​the object respectively, while paying attention to the local features obtained from the edge details and the global information extracted from the panoramic object, a fine segmentation result including the edge texture and internal area of ​​the object is obtained.

[0053] Optionally, the object segmentation result of the first image can be specifically expressed as a mask (Mask, also called segmentation mask or object segmentation mask) prediction of each object in the first image, wherein the pixel value of the mask edge pixel represents the probability that the pixel point belongs to the current object.

[0054] That is, through the method provided in the embodiment of the present application, it is possible to obtain fine mask predictions of uncountable, non-fixed-shape semantic objects (such as the sky, grass, etc.) and countable, regularly-shaped instance objects (such as people, pets, etc.) in the image, which can provide edge details of the object, such as hair strands, plant branches and leaves, and so on.

[0055] It can be understood that the method provided in the embodiment of the present application can be applied to a first image including only one object to achieve fine mask prediction of one object; or, it can be applied to a first image including multiple objects. Taking the first image including object A and object B as an example, by matching the central area of ​​object A with the edge area of ​​object A, and matching the central area of ​​object B with the edge area of ​​object B, fine mask prediction of each object is achieved.

[0056] The method provided in the embodiment of the present application can be called "panoptic matting" technology, which is to perform fine segmentation on all objects in the image and obtain a soft boundary mask of each object in the image. The fine edge means that the pixel value of the edge pixel of the mask represents the probability that the pixel belongs to the current object.

[0057] The method provided in the embodiment of the present application divides all objects in the first image into object center areas and object edge areas, and realizes the unification of global information extracted from panoramic objects and local features obtained from edge details by extracting and predicting the object center areas and object edge areas respectively, thereby achieving fine image segmentation of multiple objects.

[0058] In the embodiment of the present application, an optional implementation is provided for step S101, which may specifically include:

[0059] Step S1011: extracting a first image feature of a first image, where the first image feature includes semantic information of the first image;

[0060] In practical applications, shallow feature maps and deep feature maps can be extracted from the input first image, where the shallow feature map (also called low-level feature map) contains more detailed information of the image, such as color, edge, gradient, etc.; the deep feature map (also called high-level feature map) contains richer semantic information of the image and aggregates the detailed information of the image. In the embodiment of the present application, the deep feature map can be used as the first image feature.

[0061] Step S1012: extracting edge texture features of the first image;

[0062] In the embodiment of the present application, the edge texture feature refers to a texture-related feature that may be an edge in the first image. Compared with the central area of ​​the object, the edge area of ​​the object not only needs to include fine edges, but also needs to give a corresponding object category, so the prediction of the edge area is very challenging. In the embodiment of the present application, the edge texture features of the image are extracted and strengthened in advance to prepare for the extraction of the edge area.

[0063] Optionally, the edge texture feature includes edge detail texture and semantic information of each object.

[0064] Optionally, the shallow feature map and the deep feature map extracted from the first image may be used to extract edge texture features of the first image.

[0065] Step S1013: extracting at least one object center region and at least one object edge region from the first image based on the first image feature and the edge texture feature.

[0066] In the embodiment of the present application, based on the edge texture feature and the deep semantic feature (i.e., the first image feature), the feature representations of the edge region and the center region of the object are predicted respectively, and converted into edge part masks and center part masks. The edge region of the object requires both edge texture features to obtain detailed information and deep semantic features to obtain semantic information, while the center region of the object does not require detailed information and can only use deep semantic features.

[0067] Optionally, this step may also establish a relationship between the object edge region and the object center region to ensure that the associated object edge region and object center region are from the same object.

[0068] Based on at least one of the above embodiments, in one example, a process of a panoramic fine segmentation method is as follows: Figure 3 As shown, this may include:

[0069] Step 3.1: Perform feature extraction on the input image (i.e., the first image), and extract a shallow feature map and a deep feature map (i.e., the first image feature) therefrom;

[0070] Step 3.2: Process the extracted feature map to obtain edge texture features to prepare for subsequent edge part extraction. This step can be processed using a boundary feature attention block.

[0071] Step 3.3: Based on the edge texture feature and the deep semantic feature (i.e., the first image feature), predict at least one object center region and at least one object edge region respectively, and convert them into edge region masks and center region masks. This step can be processed using a Solid and Boundary Area Extraction Module.

[0072] Step 3.4: Optimally match and fuse the two parts to obtain a panoramic fine segmentation result (i.e., the object segmentation result of the first image). This step can be processed using a fine mask matching module (Matting Matching Block). For the first image of multiple objects, since the object center area and the object edge area may not be simply matched together, each object can be considered to obtain a global optimal solution (optimal matching result).

[0073] In the embodiment of the present application, an optional implementation is provided for step S1012, which may specifically include:

[0074] Step S10121: extracting a first texture feature of the first image, where the first texture feature is used to characterize a degree of texture change in the first image;

[0075] For the embodiment of the present application, the first texture feature is a texture-sensitive image feature, which includes edge texture feature information of each object, as well as a lot of texture feature information not related to the edge of the object (including central area texture feature information, noise texture, etc., but not limited to this).

[0076] Step S10122: performing convolution processing on the first image based on the first texture feature to obtain a second texture feature of at least one object edge region;

[0077] In the embodiment of the present application, considering that the first texture feature may be interfered by noise and irrelevant texture features, this step is used to suppress non-edge texture feature information that is not related to the edge of the object, and enhance the edge texture feature information, to obtain a second texture feature after edge texture enhancement and non-edge texture suppression.

[0078] Step S10123: Determine edge texture features of the first image based on the second texture features.

[0079] For the embodiment of the present application, after obtaining the second texture feature after edge texture enhancement and non-edge texture suppression, the semantic information of the object is added to obtain the edge texture feature of the first image.

[0080] In an embodiment of the present application, an optional implementation is provided for step S10121, which may specifically include: extracting a second image feature of the first image; and performing texture gradient calculation on the second image feature to obtain a first texture feature of the first image.

[0081] In the embodiment of the present application, considering that the texture detail features of the image are more dependent on the high-resolution feature map, the high-resolution feature map (i.e., the second image feature) is extracted to obtain the texture-sensitive image features. Figure 1 It is generally a shallow semantic feature, and the shallow feature map extracted from the first image can be directly used as the second image feature.

[0082] In the embodiment of the present application, the texture-sensitive image feature (i.e., the first texture feature) is extracted by calculating the gradient of the second image feature. When the image texture changes greatly, the gradient value is relatively large; conversely, when the image texture changes slightly, the gradient value is relatively small.

[0083] In other embodiments, other methods may also be used to extract texture-sensitive image features, which is not limited in the embodiments of the present application.

[0084] In an embodiment of the present application, an optional implementation is provided for step S10122, which may specifically include: for at least one local area of ​​the first image, based on the first texture features of the local area and the first convolution kernel, generating a second convolution kernel corresponding to the local area; based on the second convolution kernel, performing convolution processing on the corresponding local area.

[0085] In the embodiment of the present application, considering that the first texture feature is not specific to the edge region, that is, the first texture feature not only includes the texture feature of the edge region, but also includes the texture features of the non-edge region such as noise and irrelevant texture, dynamic convolution is used to suppress the texture features of the non-edge region and enhance the texture of the edge region. Compared with traditional convolution, dynamic convolution is an enhanced convolution.

[0086] Traditional convolution uses the same weighted convolution kernel to perform convolution operations at all positions in the image. The value of the convolution kernel does not change with the change of the input value content, such as Figure 4a As shown (taking a 3*3 convolution kernel as an example), the contents of the upper left corner and the lower right corner of the input image are different, but the same convolution kernel is used to directly obtain the output result.

[0087] The convolution kernel of dynamic convolution will change according to the change of the local content of the input image or feature map, that is, the convolution kernel used in different positions of the input feature map is different. By imposing supervision constraints, the dynamic convolution kernel can suppress the features of non-edge areas and enhance the features of edge areas, such as Figure 4b As shown, in non-edge areas, such as the interior of an object, the convolution kernel is similar to a Gaussian kernel, which can suppress texture features, and in the edge portion of an adjacent object, the convolution kernel is similar to an edge-preserving filter kernel, which can enhance the features of this portion. Therefore, in an embodiment of the present application, the first image is divided into at least one local area, and dynamic convolution is performed for each local area.

[0088] In the embodiment of the present application, the dynamic convolution mechanism is as follows Figure 4c As shown in the figure, the image block corresponding to each local area of ​​the first texture feature F0' is first mapped by a conversion function (Convert function) f. Optionally, f can be an operation such as softmax or normalization. The mapped value is matrix multiplied (MatrixMultiplication) with the first convolution kernel (Kernel) to obtain the second convolution kernel corresponding to this local area, and then the calculated second convolution kernel is used to convolve (Convolution) the current local area to calculate the final result of the current local area.

[0089] like Figure 4d As shown, a specific example of dynamic convolution is given when the size of the first texture feature F0' is 6x6 and the size of the dynamic convolution kernel is 3x3. Specifically, the content of the upper left corner and the lower right corner of the input image is different. According to the texture-sensitive first texture feature, the weight of the convolution kernel is dynamically adjusted instead of using only one first convolution kernel to convolve the entire feature map. For different local areas in the first texture feature, combined with the content of the local area itself, matrix multiplication is performed on the first convolution kernel to adaptively adjust the convolution kernel used in the current local area, and different dynamic convolution kernels (second convolution kernels) are calculated. Then, the dynamic convolution kernel corresponding to each local area is used to convolve it, so that the edge area and non-edge area can be treated differently, and the output results after edge texture enhancement and non-edge texture suppression are obtained.

[0090] It should be noted that Figure 4a and Figure 4d The feature maps and convolution kernels shown in the figure are for illustration only, and the specific values ​​are subject to the actual implementation. That is, the specific data of the feature maps and convolution kernels do not affect the implementation of the scheme of the present application and should not be understood as a limitation on the present application.

[0091] In the embodiment of the present application, an optional implementation is provided for step S10123, which may specifically include:

[0092] Step S501: performing convolution processing on the second texture feature based on the edge dilation convolution kernel to obtain a third texture feature;

[0093] In the embodiment of the present application, the texture feature and the second texture feature of the edge area are expanded using the edge dilation convolution kernel. Specifically, Figure 5 As shown, a convolution operation is performed on the second texture feature using an edge dilation convolution kernel. The edge dilation convolution kernel has a larger size, and as the edge dilation convolution kernel expands, the feature value becomes larger and larger, which helps to expand the predicted edge area to a certain width.

[0094] Those skilled in the art should understand that Figure 5 The edge dilation convolution kernel in the embodiment is only an example and does not constitute a limitation on the embodiments of the present application. In practical applications, those skilled in the art can set the edge dilation convolution kernel used according to actual conditions, and the embodiments of the present application do not limit it here.

[0095] Step S502: determining edge texture features of the first image based on the third texture features.

[0096] In the embodiment of the present application, after obtaining the third texture feature after edge expansion, the semantic information of the object is added to obtain the edge texture feature of the first image.

[0097] In the embodiment of the present application, after the texture feature of the edge detail texture (such as the second texture feature or the third texture feature) is included, an optional implementation method is provided for how to add the semantic information of the object to the texture feature. Specifically, step S502 may include:

[0098] Step S5021: extracting a third image feature of the first image, where the third image feature includes semantic information of the first image;

[0099] From the above introduction, it can be seen that the deep feature map extracted from the first image contains richer semantic information of the image, so the deep feature map can be used as the third image feature.

[0100] In the embodiment of the present application, the size of the third image feature can be larger than that of the first image feature. Since the third image feature focuses more on edge texture and pays attention to the detailed information of the edge than the first image feature, the third image feature can use a larger deep feature map than the first image feature.

[0101] Step S5022: using the first attention network to determine the semantic features of the first image based on the third texture features and the third image features;

[0102] That is, using the first attention network, the semantic features of the first image corresponding to the third texture features are extracted from the third image features.

[0103] Optionally, convolution and pooling operations may be performed on the third texture feature to extract Q (Query) features; convolution and pooling operations may be performed on the third image feature to extract K (Key) features and V (Value) features. The Q feature is multiplied by the K feature, and mapped through a sigmiod function, and the mapping result is multiplied by the V feature to obtain the corresponding semantic features in the third image feature.

[0104] Step S5023: Fusing the semantic feature and the third texture feature to obtain the edge texture feature of the first image.

[0105] Optionally, convolution can be used to assign semantic features obtained from the third image features to the edge extension feature map (ie, the third texture feature), thereby obtaining edge texture features that include not only edge detail features but also corresponding object semantic information.

[0106] In the embodiment of the present application, corresponding semantic features are added on the basis of the third texture features that have been obtained, so that the edge features can correspond to specific objects.

[0107] In one example, after extracting the third image feature F2 containing rich semantic information, a method of using the attention mechanism to assign the semantic information in F2 to the third texture feature (texture feature after edge expansion) can be as follows: Figure 6 As shown, this may include:

[0108] Step S6.1: performing a convolution operation (Conv) and a pooling operation (Pooling) on ​​the third texture feature to extract a Q vector;

[0109] Step S6.2: Perform convolution operation (Conv) and pooling operation (Pooling) on ​​F2 to extract K vector;

[0110] Step S6.3: Apply transpose multiplication to the Q vector and the K vector, and perform a sigmiod operation to obtain the attention weight coefficient;

[0111] Step S6.4: Perform convolution operation (Conv) and pooling operation (Pooling) on ​​F2 to extract V vector;

[0112] Step S6.5: weight the V vector according to the attention weight coefficient and output the semantic features of F2;

[0113] Step S6.6: Combine the third texture feature with the semantic feature information by multiplication to obtain an edge texture feature F having both edge texture sensitive features and semantic information. boundary .

[0114] In other embodiments, the edge expansion operation is optional, and the semantic information of the object can be directly added to the second texture feature to obtain the edge texture feature, that is, step S502 can include: extracting the third image feature of the first image, the third image feature includes the semantic information of the first image; using the first attention network, based on the second texture feature and the third image feature, determining the semantic feature of the first image; fusing the semantic feature and the second texture feature to obtain the edge texture feature of the first image. The specific processing process can be referred to steps S5021 to S5023, and similar processing processes are not repeated here.

[0115] Based on at least one of the above embodiments, in one example, the processing process of the edge feature attention module used in step 3.2 is as follows: Figure 7 As shown, the input of this module is the second image feature F0 (the shallow semantic feature from the feature extractor can be used) and the third image feature F2 (the deep semantic feature from the feature extractor can be used), which mainly includes the following process:

[0116] Step S7.1: F0 contains rich detail texture information, and the texture-sensitive image feature (ie, the first texture feature) is extracted by calculating the gradient of F0.

[0117] Step S7.2: Dynamically adjust the weight of the convolution kernel according to the first texture feature. For different local areas in the feature map, different convolution kernels are calculated according to the image content. In non-edge texture sensitive areas (such as the internal area of ​​the object), the dynamic convolution kernel can further suppress the texture, while in edge texture sensitive areas (such as the edge area of ​​the object), the dynamic convolution can further enhance the texture. Therefore, the texture detail features related to the edge of the object are highlighted and other features are suppressed.

[0118] Step S7.3: Perform an edge dilation convolution operation on the edge texture sensitive image feature (i.e., the second texture feature) that has been adjusted by dynamic convolution to achieve the expansion of the texture edge range.

[0119] Step S7.4: Calculate the correlation between the texture-sensitive image feature after edge expansion (i.e., the third texture feature) and the third image feature F2, use the third image feature F2 as a query object, inject semantic information into the third texture feature, so that the edge texture feature finally outputted can correspond to the object in the image.

[0120] In the embodiment of the present application, an optional implementation is provided for step S1013, which may specifically include:

[0121] Step S10131: extracting object feature representation of the first image;

[0122] In the embodiment of the present application, the object feature representation is used to reflect the category and spatial information of the object.

[0123] In an optional implementation, this step may specifically include: extracting a fourth image feature of the first image, the fourth image feature including semantic information of the first image; and extracting a first query feature and a second query feature of the first image from the fourth image feature as an object feature representation by initializing the query feature.

[0124] Optionally, the first query feature and the second query feature are used to extract at least one object edge region and at least one object center region, respectively.

[0125] The initialization query feature can be a set of trained query vectors, for example, it can include N query sub-queries. The initialization query feature and the fourth image feature are input into the first decoder (Transformer Decoder), and the initialization query feature extracts the category and spatial information from the fourth image feature to obtain N intermediate query sub-queries. Optionally, the N intermediate query sub-queries output by the first decoder are evenly divided into two parts, and input as the first query feature and the second query feature respectively. Among them, the N / 2 query sub-queries as the first query feature are more sensitive to edge information after learning, and the N / 2 query sub-queries as the second query feature are more sensitive to central area information after learning. In practical applications, the relationship between the first query feature, the second query feature and the query sub-query can be set and trained according to actual conditions, and the embodiments of the present application are not limited here.

[0126] Step S10132: Using a self-attention decoding network, based on the object feature representation, the first image feature and the edge texture feature, determine a regional feature representation of at least one object edge region and a regional feature representation of at least one object center region.

[0127] Specifically, a self-attention decoding network is used to update the object feature representation through the first image feature and the edge texture feature to obtain a regional feature representation of at least one object edge region and a regional feature representation of at least one object center region.

[0128] In an embodiment of the present application, in order to obtain a regional feature representation of at least one edge region of an object (which may be referred to as edge feature representation or edge embedding for short) and a regional feature representation of at least one central region of an object (which may be referred to as central feature representation, or entity feature representation, or central embedding, or entity embedding for short), a self-attention decoding network is used to predict edge embedding and entity embedding.

[0129] Since the edge texture feature combines edge information and semantic information, it can be selected to extract a regional feature representation of at least one object edge region.

[0130] Specifically, the first query feature (such as the N / 2 query sub-output by the first decoder) is combined with the edge texture feature F boundary The two are jointly input into the self-attention decoding network for interactive calculation, and the N / 2 intermediate queries will be boundary In the query, N / 2 edge feature representations (ie, updated N / 2 query sub-parts) are output, and the feature representations carry edge information and are used to represent at least one object edge region.

[0131] And the second query feature (such as the other N / 2 query sub-outputs of the first decoder) and the small-scale first image feature F1 are jointly input into the self-attention decoding network for interactive calculation, and the other N / 2 intermediate query sub-queries are queried in F1, and N / 2 central feature representations (i.e., the updated other N / 2 query sub-queries) are output, which carry entity information and are used to represent the central area of ​​at least one object.

[0132] Among them, since the edge area of ​​the object focuses on the edge texture, while the center area of ​​the object focuses on the internal area of ​​the object, not paying attention to too many detailed features, only a small feature map is sufficient. Therefore, the size of the first image feature F1 used to extract the center embedding can be smaller than the edge texture feature F used to extract the edge embedding boundary (same size as the second image feature F0 and / or the third image feature F2). Due to its small size, F1 can save network computing resources.

[0133] Optionally, the self-attention decoding network can adopt a Decoder of a Transformer model based on a self-attention mechanism, which can also be referred to as a second decoder, but is not limited to this.

[0134] Step S10133: Determine at least one object center region and at least one object edge region based on the region feature representation of at least one object edge region and the region feature representation of at least one object center region.

[0135] In the embodiment of the present application, all region feature representations (embeddings) can ensure that each object can obtain the corresponding edge region and center region.

[0136] In the embodiment of the present application, the self-attention decoding network includes at least one second attention network and at least one third attention network. For step S10132, it may further include:

[0137] Step S801: determining a first intermediate region feature representation and a first feature processing result of at least one object edge region based on the object feature representation and the edge texture feature;

[0138] Optionally, based on the first query feature (which can be understood as the Q feature of the attention mechanism) and the edge texture feature F boundary , obtain the first intermediate region feature representation (i.e., the updated first query feature) and the first feature processing result (which can be F boundary K feature and V feature).

[0139] Step S802: determining a second intermediate region feature representation and a second feature processing result of at least one object central region based on the object feature representation and the first image feature;

[0140] Optionally, based on the second query feature (which can also be understood as the Q feature of the attention mechanism) and the small-scale first image feature F1, a second intermediate region feature representation (i.e., an updated second query feature) and a second feature processing result (which can be the K feature and V feature of F1) are obtained.

[0141] Step S803: using at least one second attention network, according to the second feature processing result, updating the first intermediate region feature representation to obtain a region feature representation of at least one object edge region;

[0142] For example, the second attention network updates the second query feature (Q feature) again according to the K feature and V feature of F1.

[0143] For the embodiment of the present application, the purpose of this step is to make the edge embedding carry the corresponding central area information.

[0144] Specifically, the second feature processing result and the first middle area feature representation (which can be understood as middle edge embedding) are cross-attention processed, that is, a semantic connection between the first query feature and the second query feature is established to ensure that the edge area and the central area are mutually constrained to avoid abnormal situations.

[0145] As an example, Figure 8As shown in the figure, two objects are close together in the image. When extracting the edge area, the edges of the two objects may be extracted as the edge area of ​​one object. After establishing a semantic connection with the central area information through cross-attention, the edge area of ​​each object can be identified.

[0146] Step S804: using at least one third attention network, based on the first feature processing result, the second intermediate region feature representation is updated to obtain a regional feature representation of at least one object central region.

[0147] For example, the third attention network is based on F boundary The K feature and V feature are used to update the second query feature (Q feature) again.

[0148] Similar to step S803, cross attention is also used to establish a semantic connection between the second query feature and the first query feature, and a regional feature representation of at least one object center area is extracted through interaction.

[0149] In the embodiment of the present application, the self-attention decoding network establishes semantic connections through cross-attention to ensure that each object can obtain the query result.

[0150] In the embodiment of the present application, the regional feature representation includes at least one of the following:

[0151] (1) Class feature representation (class embedding or class prediction): used to indicate the category represented by the edge area or the center area;

[0152] (2) Identity embedding: used to represent each person’s identity information for matching;

[0153] (3) Mask feature representation (mask embedding or mask prediction): Use a dynamic convolution kernel to calculate the mask.

[0154] Optionally, after obtaining the regional feature representation of at least one edge region of an object or the regional feature representation of at least one central region of an object, the above three types of feature representations may be predicted through a multilayer perceptron (MLP) layer.

[0155] Based on at least one of the above embodiments, in one example, the processing process of the edge and center area extraction module used in step 3.3 is as follows: Fig. 9As shown in the figure, the module uses the boundary transformer decoder and the solid transformer decoder to interactively iterate the input intermediate query sub-iterations for example. The main process includes:

[0156] Step S9.1: Divide the N intermediate queries equally into two parts and input them into the edge transformer decoder and the center transformer decoder respectively.

[0157] Step S9.2: In order to obtain edge embeddings, edge embeddings are predicted in an iterative interactive manner. First, N / 2 intermediate query sub-queries (i.e., the first query feature, or the Q feature of the attention mechanism) are combined with the edge texture feature F boundary The first Transformer layer is fed into the first Transformer layer for interactive calculation, and outputs N / 2 edge feature representations (embeddings) (i.e., a first intermediate region feature representation, which can also be understood as an updated Q feature) and F boundary The K and V features of the center transformer decoder (i.e., the K and V features of F1, which can also be called the second feature processing result) are used to make the output edge feature representation (embedding) contain edge information. Then, in order to make the edge feature representation (embedding) contain the corresponding entity information, the intermediate result of the center transformer decoder (i.e., the K and V features of F1, which can also be called the second feature processing result) and the edge feature representation (embedding) of the first branch are cross-attention processed. The intermediate result of the center transformer decoder is used to update the N / 2 edge feature representations (embedding) again. The cross-attention layer outputs the updated N / 2 edge feature representations (embedding) (i.e., the updated Q features) and F boundary K features and V features. Through at least one set of transformer layers and cross attention layers ( Fig. 9 Taking two groups as an example) the final N / 2 edge feature representations (embeddings) are obtained.

[0158] Step S9.3: In order to obtain the central embedding, similar to step S9.2, the central embeddings are predicted in an interactive iterative manner. The input is N / 2 intermediate query sub-queries and the small-scale first image feature F1. After interactive iteration, N / 2 central feature representations (embeddings) are output.

[0159] Step S9.4: Pass the edge feature representation (embedding) and the center feature representation (embedding) through their respective MLPs, and the MLP predicts the above three types of vectors, namely, the category feature representation, the identity feature representation, and the mask feature representation.

[0160] The embodiment of the present application calculates edge feature representations (embeddings) and center feature representations (embeddings) of all objects in the image through feature maps of different scales to improve efficiency. All feature representations (embeddings) can ensure that each object can obtain the corresponding edge area and center area.

[0161] In the embodiment of the present application, an optional implementation is provided for step S102. Specifically, step S102 may include: matching the extracted object center area and the object edge area according to at least one of the following matching methods to obtain an object segmentation result of the first image:

[0162] (1) Direct Matching

[0163] In some scenarios, the center area of ​​the object and the edge area of ​​the object can be matched directly. For example, when the number of objects is small, they can be matched directly.

[0164] (2) Match by category

[0165] That is, the central area and the edge area of ​​objects of the same category are matched.

[0166] (3) Matching by identity

[0167] That is, the central area of ​​the object with relevant identity information (which can also be understood as having relevant semantics) and the edge area of ​​the object are matched. Identity information includes but is not limited to the central position, shape, etc. Matching can also be performed based on information such as the central position and / or shape. For example, matching based on the central position is used, that is, the central area of ​​the object corresponding to the central position and the edge area of ​​the object are matched.

[0168] In the embodiment of the present application, multiple matching methods can be combined. In an optional implementation, step S102 may specifically include:

[0169] Step S1021: determining the semantic relevance and category relevance between the extracted object center region and the object edge region;

[0170] In the embodiment of the present application, each object edge region can find one or more object center regions related to its semantics, and can also find one or more object center regions related to its category, such as Fig.10Specifically, the more corresponding the object center area and the object edge area are, the greater the correlation is.

[0171] Step S1022: matching the object center area and the object edge area based on category relevance and semantic relevance;

[0172] For the embodiments of the present application, it is possible to avoid mismatches caused by the edge area of ​​an image being applicable to the center area of ​​multiple images. It is also possible to avoid mismatches caused by the center positions of two objects being too close, such as two people standing in front and behind to take a photo. It is also possible to avoid mismatches caused by the presence of multiple objects of the same category in the picture, and it is also possible to avoid mismatches caused by inaccurate predictions of the object category.

[0173] That is, the embodiment of the present application can further ensure the accuracy of matching through the semantic correlation and category correlation between the central area of ​​the object and the edge area of ​​the object.

[0174] Step S1023: Fusing the matched object center region and object edge region to obtain an object segmentation result of the first image.

[0175] See also Figure 2 By fusing the matched object center area and object edge area, local features obtained from edge details and global information extracted from panoramic objects can be obtained, and a fine segmentation result containing the edge texture and internal area of ​​the object can be obtained.

[0176] In the embodiment of the present application, an optional implementation is provided for step S1022, which may specifically include:

[0177] Step S10221: determining the semantic correlation between the object center region and the object edge region according to the identity feature representation of the object edge region and the identity feature representation of the object center region;

[0178] Optionally, based on the identity feature representation of the edge region of the object and the identity feature representation of the central region of the object, an identity similarity matrix can be calculated to reflect the degree of semantic connection between the edge regions of different objects and the central region of the object, that is, the semantic relevance. The identity feature representation is a set of vectors representing the features of the object, such as position, shape, etc.

[0179] Optionally, the identity similarity matrix is ​​calculated by performing matrix multiplication on the identity feature representation of the edge area of ​​the object and the identity feature representation of the central area of ​​the object to generate an identity similarity matrix of size (N / 2)*(N / 2), and normalizing it through calculation to output the final identity similarity matrix.

[0180] Step S10222: Determine the category correlation between the object center region and the object edge region based on the category feature representation of the object edge region and the category feature representation of the object center region.

[0181] Optionally, based on the category feature representation of the object edge region and the category feature representation of the object center region, a category similarity matrix can be calculated to reflect the category similarity between different object edge regions and the object center region, that is, category correlation.

[0182] Optionally, the category similarity matrix is ​​calculated by calculating the similarity between the category feature representation of the edge region of the object and the category feature representation of the center region of the object to generate a category similarity matrix of size (N / 2)*(N / 2).

[0183] Furthermore, step S1022 may specifically include: determining a matching matrix between the object center region and the object edge region based on the category relevance and the semantic relevance, wherein the object edge region and the object center region in the matching matrix have a one-to-one mapping relationship;

[0184] Optionally, the calculated identity similarity matrix and category similarity matrix are superimposed together, and the superimposed similarity matrices are merged through convolution to reduce the many-to-many mapping relationship between the edge area and the central area, and generate a matching matrix between the object center area and the object edge area.

[0185] In an embodiment of the present application, in order to ensure that the object edge area and the object center area in the matching matrix have a one-to-one mapping relationship, that is, each object edge area has a unique matching object center area, the correlation matrix of the object center area and the object edge area can be determined based on category correlation and semantic correlation, and then the optimal match in the correlation matrix can be determined to obtain the matching matrix of the object center area and the object edge area.

[0186] Optionally, the calculated identity similarity matrix and category similarity matrix are superimposed together, and the superimposed similarity matrices are merged and preliminarily optimized through convolution to generate a primary matching matrix (i.e., a correlation matrix), and the primary matching matrix is ​​optimized using the Sinkhorn algorithm to determine the optimal match in the primary matching matrix, thereby generating a final matching matrix. Among them, the Sinkhorn algorithm is a mathematical method for solving the optimal transmission problem. In the embodiment of the present application, it is used to obtain the optimal matching result between the edge area and the central area. In other embodiments, other algorithms may also be used, and the embodiment of the present application is not limited here.

[0187] Furthermore, step S1023 may specifically include: based on the matching matrix, fusing the matched object center region and object edge region to obtain the object segmentation result of the first image.

[0188] That is, according to the generated matching matrix, the object center area corresponding to each object edge area can be found, and the two parts can be superimposed and fused together, that is, the masks of different edge areas and center areas are merged using the matching matrix to generate a refined segmentation result for each object in the first image.

[0189] Based on at least one of the above embodiments, in one example, the processing process of the fine mask matching module used in step 3.4 is as follows: Fig.11 As shown in the figure, this module obtains the global optimal matching result of the edge area and the central area by calculating the matching matrix with a one-to-one mapping relationship between the edge area and the central area, and matches the two areas according to the matching matrix to obtain the refined segmentation result of each object. The main process includes:

[0190] Step S11.1: The mask embeddings respectively output by the two branches of the edge and center region extraction module (i.e., the edge transformer decoder and the center transformer decoder) are used as dynamic convolution kernels to perform dynamic convolution on the image features of the first image after encoding and decoding (e.g., the third image features, which may also be referred to as pixel embeddings herein), to generate masks of the edge region and the center region respectively;

[0191] Step S11.2: In order to generate the cost matrix, the identity embeddings corresponding to the edges and centers output by the edge and center region extraction module are used for matrix multiplication, and normalized through calculation to generate the identity similarity matrix;

[0192] Step S11.3: In order to ensure the matching accuracy, the class feature representations (class embeddings) corresponding to the edge and center output by the edge and center area extraction modules are used to perform similarity calculations to generate a class similarity matrix.

[0193] Step S11.4: superimpose the two similarity matrices of step S11.3 and step S11.4, and merge and preliminarily optimize the superimposed similarity matrices through convolution to generate a primary matching matrix (i.e. the above-mentioned correlation matrix, also called the primary matching cost matrix);

[0194] Step S11.5: Based on the primary matching matrix generated in step S11.4, the primary matching matrix is ​​optimized using the Sinkhorn optimization algorithm to generate a final matching matrix, in which each element to be matched (center or edge) has a unique matching target;

[0195] Step S11.6: According to the matching matrix generated in step S11.5, find the central area corresponding to each edge area, and superimpose and merge the two parts together to generate a refined segmentation result for each defined category object in the image.

[0196] Based on at least one of the above embodiments, in one example, a complete process of a panoramic fine segmentation method is as follows: Fig.12 As shown in the figure, this module obtains the global optimal matching result of the edge area and the central area by calculating the matching matrix with a one-to-one mapping relationship between the edge area and the central area, and matches the two areas according to the matching matrix to obtain the refined segmentation result of each object. The main process includes:

[0197] Step S12.1: Input the picture to be processed (ie, the first image).

[0198] Step S12.2: Extract features from the input image. Optionally, use a feature extraction network to extract image features of the first image, wherein the feature extraction network includes a (pixel) encoder (pixel-encoder) and a (pixel) decoder (pixel-decoder). Obtain deep image features F1, F2, F3, and shallow image features F0 in the feature extraction network. Optionally, the features extracted using the first preset feature extraction layer of the decoder are used as the first image features (i.e., F1); the features extracted using the second preset feature extraction layer of the encoder are used as the second image features (i.e., F0); the features output by the decoder are used as the third image features (i.e., F2, which can also be used as pixel embedding); and the features output by the encoder are used as the fourth image features (i.e., F3). F0 contains detail information, F3 contains sufficient semantic information, and F1 and F2 contain more semantic information.

[0199] Step S12.3: Using the image features F2 and F0, with the help of the edge feature attention module, obtain the edge texture feature F boundary . Compared with the central area of ​​the object, the edge area of ​​the object not only needs to include fine edges, but also needs to give the corresponding object category. Therefore, the prediction of the edge area is more challenging than that of the central area. Among them, the edge feature attention module aims to extract and enhance the edge texture features of the target object in the image, extract the edge texture features of the edge area in advance, and prepare for the extraction of the edge area of ​​the object. The edge texture features include the edge detail texture and the semantic information of each object. The detail features rely more on the high-resolution feature map (i.e., F0). The high-resolution feature with detail features Figure 1Generally, it is a shallow image feature, which usually appears in the first few layers of the feature extraction network and will be disturbed by noise and irrelevant texture features. The edge feature attention module can dynamically adjust the weight of the convolution kernel according to the edge texture sensitive features, suppress the noise and internal texture (i.e., non-edge texture sensitive areas), enhance the edge texture (i.e., edge texture sensitive areas), and add the semantic information from the deep image features (i.e., F2) to the edge texture features through edge attention. The detailed implementation method can be found in the introduction above and will not be repeated here.

[0200] Step S12.4: Initialize a set of trained query vectors, i.e., N query sub-queries. Input the N query sub-queries and the image features (i.e., F3) output by the pixel encoder into the Transformer decoder. The N query sub-queries extract category and spatial information from the fourth image features to obtain N intermediate query sub-queries. Divide the N intermediate query sub-queries output by the Transformer decoder into two equal parts to obtain the first query feature and the second query feature.

[0201] Step S12.5: The edge texture feature F boundary and deep semantic features F1, as well as two copies of the intermediate query sub-input edge and center area extraction module, to extract the edge area and the center area respectively; the embodiment of the present application divides the segmented object in the first image into the edge area and the center area, and predicts and matches the edge area and the center area of ​​the segmented object respectively to obtain a panoramic fine segmentation mask. The purpose of the edge and center area extraction module is to predict the edge area and the center area. Among them, the edge area focuses on the edge texture, while the center area focuses on the internal area of ​​the object, does not pay attention to the detailed information of the edge, and only requires a small feature map (ie, F1). Based on the difference between the edge area and the center area, the edge transformer decoder branch and the center transformer decoder branch in the edge and center area extraction module use feature maps of different scales for the two areas (ie, F boundary At the same time, the two regions also establish a semantic connection through cross attention to ensure that each object can obtain the corresponding feature representation, which can effectively obtain the edge area and center area of ​​the target object and improve the prediction efficiency. The MLP is used to predict the category feature representation, identity feature representation and mask feature representation corresponding to the edge area and center area respectively. The detailed implementation method can be found in the above introduction and will not be repeated here.

[0202] Step S12.5: Match and fuse the edge area and the central area through the fine mask matching module, and output a panoramic fine segmentation mask. Due to direct matching, there may be a situation where one edge area corresponds to multiple central areas, that is, the obtained edge area and the central area cannot be matched one by one. The fine mask matching module is a module that matches the edge area with the central area. The category feature representation and identity feature representation obtained by the edge and central area extraction module are used to calculate the category similarity matrix and the identity similarity matrix respectively. The mutual constraints between the two are used to reduce the many-to-many matching situation, and further obtain a one-to-one matching matrix with the help of global optimization. With the guidance of the matching matrix, combined with the mask feature representation obtained by the edge and central area extraction module, the fine mask segmentation result of each object is obtained by fusion. The detailed implementation method can be found in the above introduction, which will not be repeated here.

[0203] Step S12.6: Output a multi-label global mask.

[0204] The panoramic fine segmentation method provided in the embodiment of the present application can provide a fine segmentation result of panoramic objects (uncountable semantic categories with unfixed shapes, such as the sky and grass, and countable instance categories with certain regular shapes, such as people and pets) in the image, which plays a vital role in image post-processing. For example, whether in image editing, image matting or image enhancement, fine multi-object segmentation will bring more perfect results.

[0205] As an example, the panoramic fine segmentation method provided by the embodiment of the present application can make the edges of tree branches and leaves look more real and natural in image editing. For another example, the panoramic fine segmentation method provided by the embodiment of the present application can make the hair of a person look more detailed in image cutout. For another example, the panoramic fine segmentation method provided by the embodiment of the present application can make the image have a more naturally transitioned boundary line in image enhancement.

[0206] As a basic supporting technology for image post-processing, the panoramic fine segmentation method provided in the embodiment of the present application can be applied to image processing on mobile terminals, can realize real-time panoramic fine segmentation on mobile terminals, and can be used in HDR (High Dynamic Range Imaging).

[0207] In one example, the panoramic fine segmentation method provided by the embodiment of the present application can be applied to the album function of a mobile phone. Specifically, before step S101, the method can also include the step of receiving an image segmentation trigger instruction from a user.

[0208] For example, when the user performs personalized editing on the image, the function is activated, or when the user edits the image and starts automatic optimization, the function is activated. In practical applications, those skilled in the art can set the triggering method of the image segmentation trigger instruction of the function according to the actual triggering method, and the embodiments of the present application are not limited here.

[0209] Furthermore, the panoramic fine segmentation method provided by the embodiment of the present application may further include the following steps after obtaining the object segmentation result of the first image in step S102:

[0210] Step A1: outputting an object segmentation result of the first image, where the object segmentation result includes at least one type of object segmented in the first image, and each type of object includes at least one;

[0211] When the fine image processing function is triggered, the image will be finely panorama segmented, and each object in the image will be activated and presented separately.

[0212] Step A2: receiving the user's object selection and / or object editing instruction;

[0213] That is, the user can select and / or use each object according to needs.

[0214] Step A3: In response to the object selection and / or object editing instruction, the object corresponding to the object selection and / or object editing instruction is processed accordingly.

[0215] For example, users can drag, edit and share any object; or can perform editing functions such as AI sky replacement, or can perform cutout functions, for example, users can cut out the main target of the image in the album for emoticon package production, etc.; or can perform local (intelligent) image optimization functions, image optimization will be based on the panoramic fine segmentation results (can also be based on the semantic features of different objects in the image), for at least one segmented object to optimize or enhance the image quality, such as performing personalized inpainting operations. In actual applications, those skilled in the art can set the function according to the actual usage scenario, and the embodiments of this application are not limited here.

[0216] The panoramic fine segmentation provided in the embodiment of the present application can better extract the details of the target object, and the image editing based on the panoramic fine segmentation can naturally integrate the editing results of different objects. For example, in a scene with multiple objects, the user hopes to beautify each part in a different way, such as image redrawing. Specifically, for scenes with humans, sky, and grass, one can expect a bluer sky, greener grass, women with smoother skin, and men with more hair. The panoramic fine segmentation provided in the embodiment of the present application makes it a real-time function, all parts can be personalized and beautified at the same time, and since each part is based on a mask, they can eventually be merged together without gaps, which looks more natural.

[0217] In another example, the panoramic fine segmentation method provided in the embodiment of the present application can be applied to the camera mode of a mobile phone. Specifically, before step S101, the method can also include the step of receiving a shooting trigger instruction from a user.

[0218] For example, when the user selects the mobile phone camera function, the function is activated. In practical applications, those skilled in the art can set the triggering mode of the split shooting trigger instruction of the function according to the actual situation, and the embodiment of the present application is not limited here.

[0219] Furthermore, the panoramic fine segmentation method provided by the embodiment of the present application may further include the following steps after obtaining the object segmentation result of the first image in step S102:

[0220] Step B1: determining target shooting parameters of each object in the object segmentation result of the first image;

[0221] Step B2: based on the target shooting parameters of each object, shooting corresponding second images;

[0222] Step B3: Fusing the object segmentation results of the corresponding objects in each second image to obtain a third image.

[0223] As an example, in a scene, with the best exposure parameter for the sky, person A and person B appear dark; with the best exposure parameter for person A, the sky and person B appear overexposed; with the best exposure parameter for person B, the sky appears overexposed, while person A appears not bright enough.

[0224] In the embodiment of the present application, efficient panoramic fine segmentation separates the objects in the image, and the camera can calculate the optimal exposure parameters for different objects. The device takes multiple frames of the scene according to the parameters, and fuses the objects (exposed parts) with the best exposure parameters based on the fine segmentation mask to obtain the best exposure image. Because the mask segmentation results of each object can be obtained, it is very useful for scenes with large brightness differences.

[0225] It should be emphasized that the panoramic fine segmentation technology provided in the embodiment of the present application is significantly different from various existing segmentation technologies applied to mobile terminals:

[0226] First, the objects are different. The most widely used semantic segmentation technology currently divides images according to semantic categories, but it does not have the ability to segment individual instances. For example, when there are multiple people in the scene, they cannot be distinguished. Or, the salient target segmentation launched by the album cutout function mainly focuses on a single subject in the image, and only one object is segmented. The fine segmentation function launched for detail extraction only focuses on the foreground of the image, and it can also only give an object segmentation mask. The limited segmentation range cannot meet the user's needs for personalized processing of image regions.

[0227] Second, the effects are different. The segmentation technologies currently used on mobile devices (semantic segmentation, instance segmentation, panoramic segmentation, etc.) are mainly limited to hard boundary segmentation, which will affect the accuracy of subsequent image optimization.

[0228] The panoramic fine segmentation provided by the embodiment of the present application provides a solution for multi-object fine segmentation, allowing the user to obtain each object with soft edges, which provides a guarantee for subsequent image processing. Specifically, the panoramic fine segmentation provided by the embodiment of the present application has at least one of the following advantages over the prior art:

[0229] (1) The segmentation edge is not a hard edge and more edge details can be retained.

[0230] (2) It focuses on both global information and low-level semantic features such as details, which can be extended to extract edge information of panoramic information.

[0231] (3) It can well support complex pixel-level tasks, such as image redrawing and image cutout. For example, it can optimize the sky, plants, and portraits separately at the same time.

[0232] (4) It has a global consideration for multi-label segmentation.

[0233] (5) It can focus on multiple objects at the same time and obtain masks of multiple objects at the same time.

[0234] (6) The network can be encapsulated into an integrated model, making it more efficient and convenient to use.

[0235] (7) The model execution time is short and is not affected by the number of objects in the image.

[0236] The technical solution provided in the embodiments of the present application can be applied to various electronic devices, including but not limited to mobile terminals, smart terminals, etc., such as smart phones, tablet computers, laptops, smart wearable devices (such as watches, glasses, etc.), smart speakers, vehicle terminals, personal digital assistants, portable multimedia players, navigation devices, etc., but not limited to these. It can be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the structure according to the embodiments of the present application can also be applied to fixed-type terminals, such as digital televisions, desktop computers, etc.

[0237] The technical solution provided in the embodiments of the present application can also be applied to image styles in servers, such as independent physical servers, or server clusters or distributed systems composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0238] Specifically, the technical solution provided in the embodiments of the present application can be applied to image AI editing applications or AI cameras on various electronic devices to achieve advanced image processing based on sophisticated multi-object image panoramic segmentation.

[0239] An embodiment of the present application also provides an electronic device, which includes a processor and, optionally, may also include a transceiver and / or a memory coupled to the processor, and the processor is configured to execute the steps of the method provided in any optional embodiment of the present application.

[0240] Fig.13 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present invention is shown in FIG. Fig.13 As shown, Fig.13 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0241] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0242] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0243] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0244] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0245] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0246] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0247] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0248] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0249] The above text and drawings are provided only as examples to help readers understand the present application. They are not intended to and should not be interpreted as limiting the scope of the present application in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present application, and other similar implementation means based on the technical ideas of the present application are adopted, which also fall within the protection scope of the embodiments of the present application.

Claims

1. A method performed by an electronic device, characterized in that: include: extracting at least one object center region and at least one object edge region from the first image; The extracted central area of ​​the object and the edge area of ​​the object are matched to obtain an object segmentation result of the first image.

2. The method according to claim 1, characterized in that The step of extracting at least one object center region and at least one object edge region from the first image comprises: extracting a first image feature of the first image, wherein the first image feature includes semantic information of the first image; Extracting edge texture features of the first image; At least one object center region and at least one object edge region are extracted from the first image based on the first image feature and the edge texture feature.

3. The method according to claim 2, characterized in that The step of extracting edge texture features of the first image comprises: Extracting a first texture feature of the first image, where the first texture feature is used to characterize a degree of texture change in the first image; Based on the first texture feature, performing convolution processing on the first image to obtain a second texture feature of at least one object edge region; Based on the second texture feature, an edge texture feature of the first image is determined.

4. The method according to claim 3, characterized in that The performing convolution processing on the first image based on the first texture feature includes: For at least one local area of ​​the first image, generating a second convolution kernel corresponding to the local area based on the first texture feature and the first convolution kernel of the local area; Based on the second convolution kernel, convolution processing is performed on the corresponding local area.

5. The method according to claim 3, characterized in that: The extracting a first texture feature of the first image includes: extracting a second image feature of the first image; Perform texture gradient calculation on the second image feature to obtain a first texture feature of the first image.

6. The method according to claim 3, characterized in that The step of determining the edge texture feature of the first image based on the second texture feature includes: Based on the edge dilation convolution kernel, convolution processing is performed on the second texture feature to obtain a third texture feature; Based on the third texture feature, an edge texture feature of the first image is determined.

7. The method according to claim 6, characterized in that The step of determining the edge texture feature of the first image based on the third texture feature comprises: Extracting a third image feature of the first image, wherein the third image feature includes semantic information of the first image; Determining, using a first attention network, a semantic feature of the first image based on the third texture feature and the third image feature; The semantic feature and the third texture feature are fused to obtain an edge texture feature of the first image.

8. The method according to claim 2, characterized in that: The step of extracting at least one object center region and at least one object edge region from the first image based on the first image feature and the edge texture feature comprises: extracting an object feature representation of the first image; Using a self-attention decoding network, based on the object feature representation, the first image feature, and the edge texture feature, determine a regional feature representation of at least one object edge region and a regional feature representation of at least one object center region; At least one object center region and at least one object edge region are determined based on the region feature representation of at least one object edge region and the region feature representation of at least one object center region.

9. The method according to claim 8, characterized in that The extracting the object feature representation of the first image comprises: extracting a fourth image feature of the first image, wherein the fourth image feature includes semantic information of the first image; By initializing the query feature, the first query feature and the second query feature of the first image are extracted from the fourth image feature as the object feature representation.

10. The method according to claim 8, characterized in that The self-attention decoding network includes at least one second attention network and at least one third attention network, and using the self-attention decoding network to determine, based on the object feature representation, the first image feature, and the edge texture feature, a regional feature representation of at least one object edge region and a regional feature representation of at least one object center region, includes: Determine a first intermediate region feature representation and a first feature processing result of at least one object edge region based on the object feature representation and the edge texture feature; Determining, based on the object feature representation and the first image feature, a second intermediate region feature representation and a second feature processing result of at least one object central region; Using the at least one second attention network, and according to the second feature processing result, updating the first intermediate region feature representation to obtain a region feature representation of at least one object edge region; Using the at least one third attention network, the second intermediate region feature representation is updated according to the first feature processing result to obtain a regional feature representation of at least one object central region.

11. The method according to claim 8, characterized in that The region feature representation includes at least one of the following: category feature representation, identity feature representation and mask feature representation.

12. The method according to any one of claims 1 to 11, characterized in that The step of matching the extracted object center area with the object edge area to obtain an object segmentation result of the first image includes: Determining semantic relevance and category relevance between the extracted central region of the object and the edge region of the object; Based on category relevance and semantic relevance, matching the central area of ​​the object with the edge area of ​​the object; The matched central area of ​​the object and the edge area of ​​the object are fused to obtain an object segmentation result of the first image.

13. The method according to claim 12, characterized in that The matching of the object center area and the object edge area based on category relevance and semantic relevance includes: Determining the semantic correlation between the object center region and the object edge region according to the identity feature representation of the object edge region and the identity feature representation of the object center region; According to the category feature representation of the object edge region and the category feature representation of the object center region, the category correlation between the object center region and the object edge region is determined.

14. The method according to claim 12 or 13, characterized in that Matching the object center area and the object edge area based on category relevance and semantic relevance includes: Based on the category relevance and the semantic relevance, determining a matching matrix between the central region of the object and the edge region of the object, wherein the edge region of the object and the central region of the object have a one-to-one mapping relationship in the matching matrix; The step of fusing the matched object center region and the object edge region to obtain an object segmentation result of the first image includes: Based on the matching matrix, the matched central area of ​​the object and the edge area of ​​the object are fused to obtain an object segmentation result of the first image.

15. The method according to claim 14, characterized in that The determining, based on the category correlation and the semantic correlation, a matching matrix of the object center area and the object edge area comprises: Determine a correlation matrix between the object center region and the object edge region based on the category correlation and the semantic correlation; The best match in the correlation matrix is ​​determined to obtain a matching matrix of the central area of ​​the object and the edge area of ​​the object.

16. The method according to claim 2, 5, 6 or 9, characterized in that Extracting image features of the first image includes: Extracting image features of the first image using a feature extraction network, wherein the feature extraction network includes an encoder and a decoder; Using a feature extracted by a first preset feature extraction layer of the decoder as a first image feature; Using a feature extracted by a second preset feature extraction layer of the encoder as a second image feature; Using the feature output by the decoder as a third image feature; The feature output by the encoder is used as the fourth image feature.

17. The method according to any one of claims 1 to 11, characterized in that Before extracting at least one object center region and at least one object edge region from the first image, the method further includes: Receive image segmentation trigger instructions from the user; After obtaining the object segmentation result of the first image, the method further includes: Outputting an object segmentation result of the first image, the object segmentation result including at least one type of objects segmented in the first image, and each type of object including at least one; receiving an object selection and / or an object editing instruction from a user; In response to the object selection and / or object editing instruction, the object corresponding to the object selection and / or object editing instruction is processed accordingly.

18. The method according to any one of claims 1 to 11, characterized in that Before extracting at least one object center region and at least one object edge region from the first image, the method further includes: Receive a shooting trigger command from the user; After obtaining the object segmentation result of the first image, the method further includes: determining target shooting parameters of each object in the object segmentation result of the first image; Based on the target shooting parameters of each object, capturing corresponding second images; The object segmentation results of the corresponding objects in the second images are fused to obtain a third image.

19. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 18.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 18 is implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 18 is implemented.

Images