Image processing method and device, storage medium, program product and computer equipment

Abstract

The invention discloses an image processing method and device, a storage medium, a program product and computer equipment. The method comprises the steps of determining a first image feature corresponding to a to-be-processed image; based on the first image feature, determining region classification information and image detail information corresponding to the to-be-processed image; performing association processing based on the region classification information and the image detail information to obtain context information; performing fusion processing based on the context information, the region classification information and the image detail information to obtain enhanced features; and determining a target region in the to-be-processed image based on the enhancement feature, and performing first preset processing on the target region in the to-be-processed image, so that the region recognition precision in the image can be improved, and the related processing effect of the preset processing performed on the region can be improved.

Description

Technical Field

[0001] This application relates to the field of image technology, and in particular to an image processing method, apparatus, storage medium, program product, and computer equipment. Background Technology

[0002] In the process of image processing, it is often necessary to perform special processing on specific regions of the image (such as the edge regions of certain objects in the image).

[0003] In related technologies, edge detection is often performed directly on the image to identify specific regions, and then these identified regions are subjected to corresponding special processing. However, the region identification accuracy of this method is not high, which can easily lead to the processing effect of the aforementioned special processing failing to meet user needs. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application propose an image processing method, apparatus, storage medium, program product, and computer device, which can improve the accuracy of region recognition in an image, thereby enhancing the relevant processing effects of preset processing applied to that region.

[0005] In a first aspect, embodiments of this application provide an image processing method, including: Determine the first image feature corresponding to the image to be processed; Based on the first image features, determine the region classification information and image detail information corresponding to the image to be processed; Context information is obtained by performing association processing based on the region classification information and the image detail information; Enhanced features are obtained by fusing the context information, the region classification information, and the image detail information. Based on the enhanced features, a target region in the image to be processed is determined, and a first preset processing is performed on the target region in the image to be processed.

[0006] Optionally, the first image feature includes N third image features, where N is an integer greater than 1, and the N third image features correspond to at least two resolutions; The step of determining the region classification information and image detail information corresponding to the image to be processed based on the first image features includes: The region classification information is determined based on the third image features corresponding to the lowest resolution among the at least two resolutions; The image detail information is determined based on the third image feature corresponding to the highest of the at least two resolutions.

[0007] Optionally, determining the region classification information based on the third image features corresponding to the lowest resolution among the at least two resolutions includes: For the third image feature corresponding to the lowest resolution among the at least two resolutions, extract its semantic information; Based on the semantic information, classification processing is performed to obtain the region classification information, wherein the region classification information includes the probability that one or more pixels in the image to be processed are classified as a preset region.

[0008] Optionally, the image detail information includes a detail feature map, the size of which is the same as the size of the image to be processed.

[0009] Optionally, the step of performing association processing based on the region classification information and the image detail information to obtain context information includes: Extract the information features of the region classification information and the image detail information respectively; An attention mechanism is used to correlate the extracted information features to obtain the context information.

[0010] Optionally, the fusion processing based on the context information, the region classification information, and the image detail information to obtain enhanced features includes: Based on each of the context information, the region classification information, and the image detail information, weight information for each of them is generated through a dynamic weight model. Based on the weight information, the context information, the region classification information, and the image detail information are weighted and fused to obtain the enhanced features.

[0011] Optionally, the dynamic weight model includes convolutional layers, attention networks, and a mapping sub-model; The step of generating weight information for each of the three factors—contextual information, region classification information, and image detail information—through a dynamic weight model includes: Convert each of these into a corresponding tensor; The tensor is input into the convolutional layer to obtain the convolution result output by the convolutional layer; The convolution result is input into the attention network to obtain the weighted fusion feature information output by the attention network; The weighted fusion feature information is input into the mapping sub-model to obtain the weight information of each element output by the mapping sub-model.

[0012] Optionally, the attention network includes a spatial attention subnetwork and a channel attention subnetwork; The step of inputting the convolution result into the attention network to obtain the weighted fusion feature information output by the attention network includes: The convolution result is input into the spatial attention subnetwork to obtain the position weight information output by the spatial attention subnetwork; The convolution result is input into the channel attention subnetwork to obtain the channel weight information output by the channel attention subnetwork; The attention network generates the weighted fusion feature information based on the convolution result, the position weight information, and the channel weight information.

[0013] Optionally, the mapping sub-model includes an activation function layer and a mapping network; The step of inputting the weighted fusion feature information into the mapping sub-model to obtain the weight information of each element output by the mapping sub-model includes: The weighted fusion feature information is input into the activation function layer to obtain the compressed weight information output by the activation function layer; The compressed weight information and the weighted fusion feature information are input into the mapping network to obtain the weight information of each of them output by the mapping network.

[0014] Optionally, determining the first image feature corresponding to the image to be processed includes: The image to be processed is input into the encoder to obtain N second image features output by the encoder, wherein the N second image features correspond to at least two resolutions; Based on the N second image features, N third image features contained in the first image features are determined.

[0015] Optionally, determining the N third image features contained in the first image features based on the N second image features includes: The N second image features are input into the cross-resolution fusion network to obtain the N third image features output by the cross-resolution fusion network.

[0016] Optionally, the at least two resolutions are four resolutions, and the highest resolution among the at least two resolutions is equal to the resolution of the image to be processed.

[0017] Optionally, the first preset process includes at least one of the following: Image cutout processing; Background replacement processing.

[0018] Optionally, the method further includes: Based on the processing result of the first preset processing, the image to be processed is subjected to a second preset processing, wherein the second preset processing includes at least one of the following: classification processing, retrieval processing, and display processing.

[0019] Secondly, embodiments of this application provide an image processing apparatus, including: The first image feature determination module is used to determine the first image features corresponding to the image to be processed; The first image feature processing module is used to determine the region classification information and image detail information corresponding to the image to be processed based on the first image features. The association processing module is used to perform association processing based on the region classification information and the image detail information to obtain context information; The fusion processing module is used to perform fusion processing based on the context information, the region classification information, and the image detail information to obtain enhanced features; The first preset processing module is used to determine the target region in the image to be processed based on the enhanced features, and to perform a first preset processing on the target region in the image to be processed.

[0020] Optionally, the first image feature includes N third image features, where N is an integer greater than 1, and the N third image features correspond to at least two resolutions; The step of determining the region classification information and image detail information corresponding to the image to be processed based on the first image features includes: The region classification information is determined based on the third image features corresponding to the lowest resolution among the at least two resolutions; The image detail information is determined based on the third image feature corresponding to the highest of the at least two resolutions.

[0021] Optionally, the step of performing association processing based on the region classification information and the image detail information to obtain context information includes: Extract the information features of the region classification information and the image detail information respectively; An attention mechanism is used to correlate the extracted information features to obtain the context information.

[0022] Optionally, the fusion processing based on the context information, the region classification information, and the image detail information to obtain enhanced features includes: Based on each of the context information, the region classification information, and the image detail information, weight information for each of them is generated through a dynamic weight model. Based on the weight information, the context information, the region classification information, and the image detail information are weighted and fused to obtain the enhanced features.

[0023] Optionally, determining the first image feature corresponding to the image to be processed includes: The image to be processed is input into the encoder to obtain N second image features output by the encoder, wherein the N second image features correspond to at least two resolutions; Based on the N second image features, N third image features contained in the first image features are determined.

[0024] Optionally, the first preset process includes at least one of the following: Image cutout processing; Background replacement processing.

[0025] Optionally, the device further includes: The second preset processing module is used to perform a second preset processing on the image to be processed based on the processing result of the first preset processing, wherein the second preset processing includes at least one of the following: classification processing, retrieval processing, and display processing.

[0026] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0027] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above-described embodiments.

[0028] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0029] In summary, the embodiments of this application have at least the following beneficial effects: By employing the embodiments of this application, a first image feature corresponding to the image to be processed is determined; based on the first image feature, region classification information and image detail information corresponding to the image to be processed are determined; based on the region classification information and the image detail information, association processing is performed to obtain context information; based on the context information, the region classification information, and the image detail information, fusion processing is performed to obtain enhanced features; based on the enhanced features, a target region in the image to be processed is determined; and a first preset processing is performed on the target region in the image to be processed. In this way, image detail information and preliminary region classification information for the image can be obtained by utilizing image features, and context information can be obtained by combining the region classification information and the image detail information, thereby fusing to obtain enhanced features with richer information content, so as to achieve more accurate image region recognition, thereby improving the region recognition accuracy in the image and improving the related processing effect of the preset processing performed on the region. Attached Figure Description

[0030] Figure 1 This is a schematic flowchart of the image processing method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the feature processing provided in the embodiments of this application; Figure 3 This is a schematic diagram of feature weighted fusion provided in an embodiment of this application; Figure 4 This is a schematic diagram of the overall network for image processing provided in the embodiments of this application; Figure 5 This is a schematic diagram of model training provided in an embodiment of this application; Figure 6 This is a schematic diagram of the image processing apparatus provided in the embodiments of this application; Figure 7 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments / examples are only a part of the embodiments / examples of this application, and not all of the embodiments / examples. Based on the embodiments / examples in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0032] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. In the description of this application, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment / example" means "at least one embodiment / example"; the term "another embodiment / example" means "at least one additional embodiment / example"; the term "some embodiments / examples" means "at least some embodiments / examples."

[0033] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0034] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0035] Firstly, see [the following] Figure 1 The diagram shows a flowchart of an image processing method provided in an embodiment of this application. The image processing method can be applied to a computer device with data processing capabilities (such as a terminal device, cloud disk server, cloud, etc.). The method includes S101-S105, as follows.

[0036] S101, Determine the first image feature corresponding to the image to be processed.

[0037] In some examples, the first image feature may be an image feature obtained by calling a general image encoder to encode the image to be processed.

[0038] In some examples, when the image processing method is executed by a cloud disk server or the cloud, the image to be processed can be an image uploaded by the user. For example, it can be an image uploaded to the cloud disk server or the cloud by the user through a software application (such as a cloud disk application) while using the terminal device.

[0039] In some examples, when the image processing method is executed by a terminal device, the image to be processed may be an image captured by the user through the terminal device, and / or, the image to be processed may be an image downloaded by the user from the network through the terminal device, and / or, the image to be processed may be an image received by the user from another device through the terminal device. This application embodiment does not specifically limit this.

[0040] S102, based on the first image features, determine the region classification information and image detail information corresponding to the image to be processed.

[0041] In some examples, region classification information can be obtained by classifying the first image features into foreground, background, and / or transition regions. This allows the region classification information to contain the probability that one or more pixels in the image to be classified into a foreground, background, and / or transition region. Furthermore, the region with the highest probability for each pixel can be selected as the region to which that pixel is classified. In some cases, the region classification information may also include the pixel coordinates of the foreground region, background region, and / or transition region in the image to be processed, where the pixel coordinates of each region can be determined by the region to which each pixel is classified.

[0042] In some examples, edge detection can be performed on the image to be processed based on the first image features to obtain the edge detail information of the image to be processed, and then image detail information containing the edge detail information can be generated.

[0043] S103, perform association processing based on the region classification information and the image detail information to obtain context information.

[0044] In some examples, attention mechanisms can be used to associate the classification information of the region with the detailed information of the image, thereby generating contextual information.

[0045] In some examples, this association processing can be used to associate information related to each pixel in the image detail information with information related to each pixel in the region classification information, thereby completing the association between the two types of information, so that the context information can contain the information generated after the association between the two types of information.

[0046] S104, based on the context information, the region classification information and the image detail information, a fusion process is performed to obtain enhanced features.

[0047] In some examples, step S104 may include fusing the context information, region classification information, and image detail information to obtain the fused enhanced features.

[0048] In some examples, the aforementioned contextual information, region classification information, and image detail information can all be represented by features (such as feature vectors, feature matrices, etc.). Thus, this fusion process can be used to concatenate the contextual information, region classification information, and image detail information to obtain concatenated features, and then generate fused enhanced features based on these concatenated features (for example, directly using the concatenated features as fused enhanced features).

[0049] S105, based on the enhanced features, determine the target region in the image to be processed, and perform a first preset processing on the target region in the image to be processed.

[0050] In some examples, the first preset processing can be a special image processing associated with a target region. For example, the target region may include at least one of a foreground region, a background region, and a transition region. Here, each of the foreground region, background region, and transition region can be pre-configured to be associated with at least one special image processing. In this way, special image processing associated with that region can be performed on each region of the foreground region, background region, and / or transition region contained in the image to be processed. For example, the foreground region can be associated with special image processing such as portrait beautification and enhancement, the background region can be associated with special image processing such as background blurring, background replacement, and background compositing, and the transition region can be associated with special image processing such as compositing optimization of semi-transparent objects (hair strands, glass, smoke, etc.) and adjustment of lighting consistency.

[0051] In some examples, this enhanced feature can be used as a basis for edge detection of the image to be processed, in order to identify the target region in the image to be processed.

[0052] In one optional implementation, determining the first image feature corresponding to the image to be processed includes: The image to be processed is input into the encoder to obtain N second image features output by the encoder, wherein the N second image features correspond to at least two resolutions; Based on the N second image features, N third image features contained in the first image features are determined.

[0053] In some examples, the encoder may include a shared encoder, HRNet.

[0054] In some examples, determining the N third image features contained in the first image features based on the N second image features may include: directly determining the N second image features as the N third image features.

[0055] In this embodiment, a pre-configured encoder can be invoked to encode the image to be processed according to the at least two resolutions, generating N second image features corresponding to the at least two resolutions, thereby obtaining image features at different resolutions. Since the N third image features are generated from the N second image features, it is easy to understand that the N third image features can also correspond to the at least two resolutions. Furthermore, each of the at least two resolutions can correspond to one or more of the N third image features. Different third image features corresponding to the same resolution can be obtained by performing different encoding processes on the image to be processed. These different encoding processes can refer to different parameters other than resolution and / or different regions targeted by the encoding.

[0056] In one optional implementation, determining the N third image features contained in the first image features based on the N second image features includes: The N second image features are input into the cross-resolution fusion network to obtain the N third image features output by the cross-resolution fusion network.

[0057] In some examples, the cross-resolution converged network may include Exchange Blocks.

[0058] In some examples, the N third image features can correspond one-to-one with the N second image features, and each third image feature can be obtained by cross-resolution fusion of the corresponding second image feature and at least one of the remaining second image features.

[0059] In this embodiment, the interaction of multi-scale information among the N second image features can be enhanced by performing cross-resolution fusion on the N second image features, so that the N third image features can contain the multi-scale information obtained through interaction.

[0060] In one alternative implementation, the at least two resolutions are four resolutions, and the highest resolution among the at least two resolutions is equal to the resolution of the image to be processed.

[0061] In some examples, the four resolutions can be the original image resolution, half of the original image resolution, one-quarter of the original image resolution, and one-eighth of the original image resolution. Here, the original image resolution refers to the resolution of the image to be processed. Thus, the highest resolution refers to the original image resolution.

[0062] In one optional implementation, the first image feature includes N third image features, where N is an integer greater than 1, and the N third image features correspond to at least two resolutions; The step of determining the region classification information and image detail information corresponding to the image to be processed based on the first image features includes: The region classification information is determined based on the third image features corresponding to the lowest resolution among the at least two resolutions; The image detail information is determined based on the third image feature corresponding to the highest of the at least two resolutions.

[0063] In some examples, the third image features corresponding to the lowest resolution can be processed using S-Branch (semantic perception branch) to obtain region classification information.

[0064] In some examples, each of the at least two resolutions may correspond to one or more of the N third image features, so that each third image feature can correspond to a resolution, and the number of third image features corresponding to the lowest / highest resolution may also be one or more. The region classification information may be determined by the third image features corresponding to the lowest resolution, and the image detail information may be determined by the third image features corresponding to the highest resolution.

[0065] In some examples, the at least two resolutions can be specifically N resolutions, and the N third image features can correspond one-to-one with the N resolutions. In this way, a third image feature can be obtained for each resolution.

[0066] In some examples, the classification information of the region can be obtained by classifying the third image features corresponding to the lowest resolution into foreground regions, background regions, and / or transition regions. In this way, the classification information of the region can contain the probability that one or more pixels in the image to be processed are classified into the foreground region, background region, and / or transition region. Furthermore, the region with the highest probability for each pixel can be selected as the region to which the pixel is classified.

[0067] In some examples, edge detection can be performed on the image to be processed based on the third image features corresponding to the highest resolution to obtain the edge detail information of the image to be processed, and then generate image detail information containing the edge detail information.

[0068] In this embodiment, the third image feature corresponding to the lowest resolution has high-level semantic information suitable for global region classification, which can improve the reliability of global region classification. The third image feature corresponding to the highest resolution has original detail information suitable for fine boundary reconstruction and / or local enhancement, which can generate rich image details.

[0069] In one optional implementation, determining the region classification information based on the third image features corresponding to the lowest resolution among the at least two resolutions includes: For the third image feature corresponding to the lowest resolution among the at least two resolutions, extract its semantic information; Based on the semantic information, classification processing is performed to obtain the region classification information, wherein the region classification information includes the probability that one or more pixels in the image to be processed are classified as a preset region.

[0070] In some examples, a semantic information extractor can be invoked to extract semantic information from third-party image features.

[0071] In some examples, the classification process can be three-classified, which can be used to classify the semantic information into foreground regions, background regions, and / or transition regions. The preset regions can include foreground regions, background regions, and / or transition regions.

[0072] In some examples, the third image features corresponding to the lowest resolution can be input into the S-Branch (semantic perception branch). The semantic perception branch (e.g., MobileNetV3) extracts semantic information from the third image features and performs classification processing based on the semantic information to output the classification information of the region.

[0073] In some examples, the minimum resolution can be 1 / 8 of the original image resolution, where the original image resolution refers to the resolution of the image to be processed.

[0074] In one alternative implementation, the image detail information includes a detail feature map, the size of which is the same as the size of the image to be processed.

[0075] In some examples, the detail feature map can be a detail feature map of a transition region in the image to be processed.

[0076] In some examples, the detail feature map can be obtained by processing the third image feature corresponding to the highest resolution using D-Branch (detail-preserving branch).

[0077] In this embodiment, as much of the original detail as possible can be preserved by processing the third image feature corresponding to the highest resolution, thereby improving the processing accuracy of subsequent image processing.

[0078] In one optional implementation, the step of performing association processing based on the region classification information and the image detail information to obtain context information includes: Extract the information features of the region classification information and the image detail information respectively; An attention mechanism is used to correlate the extracted information features to obtain the context information.

[0079] In some examples, G-Branch (context-guided branch) can be used to extract information features from region classification information and image detail information respectively, and an attention mechanism can be used to associate the extracted information features to obtain context information. Compared with related technologies that directly concatenate and convolve the extracted information features to output an alpha mask, since the branches are isomorphic convolutional stacks and the fusion method is fixed concatenation and convolution, it is impossible to dynamically adjust the contribution weight of each branch according to the image content. However, the G-Branch (context-guided branch) of this embodiment can dynamically adjust the weight of different information features. For example, for hair areas where the contribution of detail branches needs to be strengthened, this embodiment can dynamically increase the weight of the corresponding features in that area. For large foreground areas where semantic branches need to be strengthened, this embodiment can dynamically reduce the noise interference of detail branches to avoid mask blurring as much as possible.

[0080] In some examples, the encoder provided in any embodiment of this application can be used to extract features from region classification information and image detail information to obtain their respective information features.

[0081] In some examples, the attention mechanism may include cross-attention mechanisms, etc., and the embodiments of this application are not limited to this.

[0082] In some examples, the context information may include a weight graph, which can be used to assist in subsequent fusion processing.

[0083] In this embodiment, an attention mechanism can be used to dynamically and accurately associate the information features of region classification information and image detail information.

[0084] In one optional implementation, the fusion processing based on the context information, the region classification information, and the image detail information to obtain enhanced features includes: Based on each of the context information, the region classification information, and the image detail information, weight information for each of them is generated through a dynamic weight model. Based on the weight information, the context information, the region classification information, and the image detail information are weighted and fused to obtain the enhanced features.

[0085] In some examples, the MDWF (Multi-scale Dynamic Weighted Fusion) module can be used to generate weighted information for each of the three elements: contextual information, region classification information, and image detail information. For example, see [link to example]. Figure 2 The MDWF module can include three identical branches (MDWF branch 1, MDWF branch 2, and MDWF branch 3 in the figure). Each of these three identical branches can be used to process one of the following: context information, region classification information, and image detail information, in order to obtain the weight information of that branch.

[0086] For example, each of the three identical branches mentioned above may include a convolutional layer, an attention network, a sigmoid function, or a fully connected MLP (Multilayer Perceptron) layer.

[0087] It is understandable that the data processing flow executed within each of the three identical branches can be the same, and the only difference is that the input data of the different branches are different.

[0088] In this embodiment, different types of features (semantics / details / context) can be standardized by using branches with the same structure to improve the fairness of subsequent weight comparisons.

[0089] In one optional implementation, the dynamic weight model includes convolutional layers, an attention network, and a mapping sub-model; The step of generating weight information for each of the three factors—contextual information, region classification information, and image detail information—through a dynamic weight model includes: Convert each of these into a corresponding tensor; The tensor is input into the convolutional layer to obtain the convolution result output by the convolutional layer; The convolution result is input into the attention network to obtain the weighted fusion feature information output by the attention network; The weighted fusion feature information is input into the mapping sub-model to obtain the weight information of each element output by the mapping sub-model.

[0090] In some examples, this tensor can be a three-dimensional tensor, which can be represented as... Where H represents height, W represents width, and C represents the number of channels.

[0091] In some examples, the convolutional layer can employ a depthwise separable convolutional kernel ( This convolutional layer can compress the number of channels of the tensor to [amount missing]. Output the convolution result .

[0092] In some examples, the mapping sub-model may include a Sigmoid function and a fully connected MLP layer connected in sequence.

[0093] In one optional implementation, the attention network includes a spatial attention subnetwork and a channel attention subnetwork; The step of inputting the convolution result into the attention network to obtain the weighted fusion feature information output by the attention network includes: The convolution result is input into the spatial attention subnetwork to obtain the position weight information output by the spatial attention subnetwork; The convolution result is input into the channel attention subnetwork to obtain the channel weight information output by the channel attention subnetwork; The attention network generates the weighted fusion feature information based on the convolution result, the position weight information, and the channel weight information.

[0094] In some examples, this positional weight information can be represented as The channel weight information can be represented as Thus, it is possible to... This is used to represent the weighted fusion feature information.

[0095] In one optional implementation, the mapping sub-model includes an activation function layer and a mapping network; The step of inputting the weighted fusion feature information into the mapping sub-model to obtain the weight information of each element output by the mapping sub-model includes: The weighted fusion feature information is input into the activation function layer to obtain the compressed weight information output by the activation function layer; The compressed weight information and the weighted fusion feature information are input into the mapping network to obtain the weight information of each of them output by the mapping network.

[0096] In some examples, this activation function layer can be a sigmoid function layer, which can be used to compress the attention weights in the weighted fused feature information to... The range, this compression weight information can be represented as .

[0097] See in some examples Figure 3 The mapping network may include, in sequence, an MLP layer, a global average pooling (GAP) layer, a fully connected layer, and a normalization layer (e.g., a Softmax normalization layer). The output of the MLP layer is... The GAP layer can be used to perform global average pooling on the output of the MLP layer to obtain the pooling result. The fully connected layer can map the pooling result to a scalar. The normalization layer can be used to perform Softmax normalization on the scalar to obtain the weight information of each and output it.

[0098] Furthermore, this enhancement feature The formulaic expression can include:

[0099] in, , , These represent contextual information, region classification information, and image detail information, respectively. It is a collection of weighted information from contextual information, region classification information, and image detail information. Furthermore, since the three identical branches have the same structure, the output... It has comparable dimensions, which facilitates generation through fully connected layers and normalized layers. To satisfy the following formula .

[0100] Correspondingly, if the structure is heterogeneous, the weight allocation may be affected by the complexity of the branches rather than purely by the importance of the content. Therefore, the three identical branches used in this embodiment can reduce design deviations.

[0101] In summary, the embodiments of this application, through a three-level architecture of "functional heterogeneity → processing homogeneity → dynamic weighting," can solve the problems of feature redundancy and fusion rigidity in related technologies, such as "homogeneous branches of MODNet and unidirectional guidance of PP-Matting," ultimately achieving a dual improvement in accuracy and efficiency for the aforementioned first preset processing (such as image matting). Furthermore, compared to the shortcomings of related technologies that rely on Trimap, the embodiments of this application do not rely on Trimap, achieving a Trimap-Free technical effect.

[0102] In some examples, Examples can be found in Table 1 below.

[0103] Table 1

[0104] In one optional implementation, the first preset process includes at least one of the following: Image cutout processing; Background replacement processing.

[0105] In this embodiment, for complex backgrounds (such as human hair or transparent objects), the accuracy and precision of image cutout and background replacement can be maintained.

[0106] In some examples, the background replacement process may include background image replacement for product images in e-commerce scenarios (the background image of the product image may also be a blank background) to improve product listing efficiency; and / or, the background replacement process may include background replacement in live video / live streaming editing and special effects compositing.

[0107] In some examples, the background removal process can include rapid background removal in short video / live streaming editing and special effects compositing.

[0108] In an optional implementation, the method further includes: Based on the processing result of the first preset processing, the image to be processed is subjected to a second preset processing, wherein the second preset processing includes at least one of the following: classification processing, retrieval processing, and display processing.

[0109] In this embodiment, when the image processing method is applied to the cloud, for example when a user uploads an image to the cloud for storage, the cloud can perform efficient and accurate classification, retrieval, and / or display processing on the uploaded image (with user authorization), greatly improving the user experience.

[0110] In some examples, in step S105, determining the target region in the image to be processed based on the enhanced features may include: generating a mask based on the enhanced features, and determining the target region in the image to be processed based on the mask. The target region may include a foreground region, a background region, and / or a transition region. The mask may be a binary or grayscale image with the same size as the original image, but each pixel value indicates whether the position belongs to a certain target region.

[0111] For example, see Figure 4The mask can be an alpha mask, which is a special type of mask with pixel values ​​ranging from [0, 1]. Here, 0 can represent complete transparency (generally representing the background area), 1 can represent complete opacity (generally representing the foreground area), and values ​​between 0 and 1 can represent semi-transparency (generally representing the transition area, such as hair strands or glass). In this way, the mask can be used to accurately describe the boundary and transparency between the foreground and the background.

[0112] Referring to the above-mentioned related embodiments, see Figure 5 Furthermore, this application embodiment also provides the loss functions used in the training process of the above-mentioned models / branches / networks, as detailed below.

[0113] Semantic loss ( ): S-Branch's three-class cross-entropy loss can be used to supervise semantic segmentation (foreground, background, transition region).

[0114] Loss of detail ( ): The L1 loss plus gradient loss in the transition region of D-Branch can optimize edge details.

[0115] Masking loss ( ): The final Laplacian loss of the alpha mask can be used to supervise the overall matting quality.

[0116] Consistency loss ( ): The transition region feature alignment loss and self-supervised loss can be used to calculate the feature differences between semantic branches and detail branches in the transition region.

[0117] Furthermore, if the above models / branches / networks are integrated into a single overall model, the overall loss function of this overall model... It can be:

[0118] in, This represents the S-Branch three-class classification cross-entropy loss. This represents the sum of the L1 loss and the gradient loss in the D-Branch transition region. This represents the Laplacian loss of the alpha mask. Indicates the feature alignment loss in the transition region ( ).

[0119] The above scheme uses a weighted caching mechanism and a sparse attention acceleration mechanism for inference optimization. The final result is a 40% reduction in dynamic weight calculation and a 3.8-fold increase in 4K image processing speed, specifically as follows: A scene feature database is established, and historical data is reused for similar scenes. Value; reduces the number of dynamic weight calculations, improving inference speed by 40%; replaces global attention with local window attention; computational complexity can be reduced from Down to ( (Window size).

[0120] Secondly, correspondingly, this application also provides an image processing apparatus capable of implementing all the processes of the image processing method provided in the above embodiments.

[0121] See Figure 6 The diagram shows a schematic representation of an image processing apparatus 600 provided in an embodiment of this application. The image processing apparatus 600 includes: The first image feature determination module 601 is used to determine the first image features corresponding to the image to be processed; The first image feature processing module 602 is used to determine the region classification information and image detail information corresponding to the image to be processed based on the first image features. The association processing module 603 is used to perform association processing based on the region classification information and the image detail information to obtain context information; The fusion processing module 604 is used to perform fusion processing based on the context information, the region classification information and the image detail information to obtain enhanced features; The first preset processing module 605 is used to determine the target region in the image to be processed based on the enhanced features, and to perform a first preset processing on the target region in the image to be processed.

[0122] In one optional implementation, the first image feature includes N third image features, where N is an integer greater than 1, and the N third image features correspond to at least two resolutions; The step of determining the region classification information and image detail information corresponding to the image to be processed based on the first image features includes: The region classification information is determined based on the third image features corresponding to the lowest resolution among the at least two resolutions; The image detail information is determined based on the third image feature corresponding to the highest of the at least two resolutions.

[0123] In one optional implementation, determining the region classification information based on the third image features corresponding to the lowest resolution among the at least two resolutions includes: For the third image feature corresponding to the lowest resolution among the at least two resolutions, extract its semantic information; Based on the semantic information, classification processing is performed to obtain the region classification information, wherein the region classification information includes the probability that one or more pixels in the image to be processed are classified as a preset region.

[0124] In one alternative implementation, the image detail information includes a detail feature map, the size of which is the same as the size of the image to be processed.

[0125] In one optional implementation, the step of performing association processing based on the region classification information and the image detail information to obtain context information includes: Extract the information features of the region classification information and the image detail information respectively; An attention mechanism is used to correlate the extracted information features to obtain the context information.

[0126] In one optional implementation, the fusion processing based on the context information, the region classification information, and the image detail information to obtain enhanced features includes: Based on each of the context information, the region classification information, and the image detail information, weight information for each of them is generated through a dynamic weight model. Based on the weight information, the context information, the region classification information, and the image detail information are weighted and fused to obtain the enhanced features.

[0127] In one optional implementation, the dynamic weight model includes convolutional layers, an attention network, and a mapping sub-model; The step of generating weight information for each of the three factors—contextual information, region classification information, and image detail information—through a dynamic weight model includes: Convert each of these into a corresponding tensor; The tensor is input into the convolutional layer to obtain the convolution result output by the convolutional layer; The convolution result is input into the attention network to obtain the weighted fusion feature information output by the attention network; The weighted fusion feature information is input into the mapping sub-model to obtain the weight information of each element output by the mapping sub-model.

[0128] In one optional implementation, the attention network includes a spatial attention subnetwork and a channel attention subnetwork; The step of inputting the convolution result into the attention network to obtain the weighted fusion feature information output by the attention network includes: The convolution result is input into the spatial attention subnetwork to obtain the position weight information output by the spatial attention subnetwork; The convolution result is input into the channel attention subnetwork to obtain the channel weight information output by the channel attention subnetwork; The attention network generates the weighted fusion feature information based on the convolution result, the position weight information, and the channel weight information.

[0129] In one optional implementation, the mapping sub-model includes an activation function layer and a mapping network; The step of inputting the weighted fusion feature information into the mapping sub-model to obtain the weight information of each element output by the mapping sub-model includes: The weighted fusion feature information is input into the activation function layer to obtain the compressed weight information output by the activation function layer; The compressed weight information and the weighted fusion feature information are input into the mapping network to obtain the weight information of each of them output by the mapping network.

[0130] In one optional implementation, determining the first image feature corresponding to the image to be processed includes: The image to be processed is input into the encoder to obtain N second image features output by the encoder, wherein the N second image features correspond to at least two resolutions; Based on the N second image features, N third image features contained in the first image features are determined.

[0131] In one optional implementation, determining the N third image features contained in the first image features based on the N second image features includes: The N second image features are input into the cross-resolution fusion network to obtain the N third image features output by the cross-resolution fusion network.

[0132] In one alternative implementation, the at least two resolutions are four resolutions, and the highest resolution among the at least two resolutions is equal to the resolution of the image to be processed.

[0133] In one optional implementation, the first preset process includes at least one of the following: Image cutout processing; Background replacement processing.

[0134] In one alternative embodiment, the device further includes: The second preset processing module is used to perform a second preset processing on the image to be processed based on the processing result of the first preset processing, wherein the second preset processing includes at least one of the following: classification processing, retrieval processing, and display processing.

[0135] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0136] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above-described embodiments.

[0137] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0138] See Figure 7 The computer device in this embodiment includes a processor 701, a memory 702, and a computer program, such as an image processing program, stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program, it implements the steps in the various image processing method embodiments described above, for example... Figure 1 The steps S101-S105 are shown.

[0139] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 702 and executed by the processor 701 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0140] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0141] The processor 701 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or processor 701 can be any conventional processor. The processor 701 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0142] The memory 702 can be used to store the computer programs and / or modules. The processor 701 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 702 and calling the data stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0143] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a non-transitory computer-readable storage medium. When the computer program is executed by the processor 701, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0144] In summary, the embodiments of this application have at least the following beneficial effects: By employing the embodiments of this application, a first image feature corresponding to the image to be processed is determined; based on the first image feature, region classification information and image detail information corresponding to the image to be processed are determined; based on the region classification information and the image detail information, association processing is performed to obtain context information; based on the context information, the region classification information, and the image detail information, fusion processing is performed to obtain enhanced features; based on the enhanced features, a target region in the image to be processed is determined; and a first preset processing is performed on the target region in the image to be processed. In this way, image detail information and preliminary region classification information for the image can be obtained by utilizing image features, and context information can be obtained by combining the region classification information and the image detail information, thereby fusing to obtain enhanced features with richer information content, so as to achieve more accurate image region recognition, thereby improving the region recognition accuracy in the image and improving the related processing effect of the preset processing performed on the region.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware platforms, or it can be implemented entirely by hardware. Based on this understanding, all or part of the technical solutions of this application that contribute to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0146] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, include: Determine the first image feature corresponding to the image to be processed; Based on the first image features, determine the region classification information and image detail information corresponding to the image to be processed; Context information is obtained by performing association processing based on the region classification information and the image detail information; Enhanced features are obtained by fusing the context information, the region classification information, and the image detail information. Based on the enhanced features, a target region in the image to be processed is determined, and a first preset processing is performed on the target region in the image to be processed.

2. The method according to claim 1, characterized in that, The first image feature includes N third image features, where N is an integer greater than 1, and the N third image features correspond to at least two resolutions; The step of determining the region classification information and image detail information corresponding to the image to be processed based on the first image features includes: The region classification information is determined based on the third image features corresponding to the lowest resolution among the at least two resolutions; The image detail information is determined based on the third image feature corresponding to the highest of the at least two resolutions.

3. The method according to claim 2, characterized in that, The determination of the region classification information based on the third image features corresponding to the lowest resolution among the at least two resolutions includes: For the third image feature corresponding to the lowest resolution among the at least two resolutions, extract its semantic information; Based on the semantic information, classification processing is performed to obtain the region classification information, wherein the region classification information includes the probability that one or more pixels in the image to be processed are classified as a preset region.

4. The method according to claim 2, characterized in that, The image detail information includes a detail feature map, the size of which is the same as the size of the image to be processed.

5. The method according to claim 1, characterized in that, The process of associating the region classification information and the image detail information to obtain context information includes: Extract the information features of the region classification information and the image detail information respectively; An attention mechanism is used to correlate the extracted information features to obtain the context information.

6. The method according to claim 1, characterized in that, The enhanced features are obtained by fusing the context information, the region classification information, and the image detail information, including: Based on each of the context information, the region classification information, and the image detail information, a dynamic weight model is used to generate weight information for each of them. Based on the weight information, the context information, the region classification information, and the image detail information are weighted and fused to obtain the enhanced features.

7. The method according to claim 6, characterized in that, The dynamic weight model includes convolutional layers, attention networks, and a mapping sub-model. The step of generating weight information for each of the three factors—contextual information, region classification information, and image detail information—through a dynamic weight model includes: Convert each of these into a corresponding tensor; The tensor is input into the convolutional layer to obtain the convolution result output by the convolutional layer; The convolution result is input into the attention network to obtain the weighted fusion feature information output by the attention network; The weighted fusion feature information is input into the mapping sub-model to obtain the weight information of each element output by the mapping sub-model.

8. The method according to claim 7, characterized in that, The attention network includes a spatial attention subnetwork and a channel attention subnetwork; The step of inputting the convolution result into the attention network to obtain the weighted fusion feature information output by the attention network includes: The convolution result is input into the spatial attention subnetwork to obtain the position weight information output by the spatial attention subnetwork; The convolution result is input into the channel attention subnetwork to obtain the channel weight information output by the channel attention subnetwork; The attention network generates the weighted fusion feature information based on the convolution result, the position weight information, and the channel weight information.

9. The method according to claim 7, characterized in that, The mapping sub-model includes an activation function layer and a mapping network; The step of inputting the weighted fusion feature information into the mapping sub-model to obtain the weight information of each element output by the mapping sub-model includes: The weighted fusion feature information is input into the activation function layer to obtain the compressed weight information output by the activation function layer; The compressed weight information and the weighted fusion feature information are input into the mapping network to obtain the weight information of each of them output by the mapping network.

10. The method according to any one of claims 1-9, characterized in that, The determination of the first image feature corresponding to the image to be processed includes: The image to be processed is input into the encoder to obtain N second image features output by the encoder, wherein the N second image features correspond to at least two resolutions; Based on the N second image features, N third image features contained in the first image features are determined.

11. The method according to claim 10, characterized in that, The step of determining N third image features contained in the first image features based on the N second image features includes: The N second image features are input into the cross-resolution fusion network to obtain the N third image features output by the cross-resolution fusion network.

12. The method according to claim 10, characterized in that, The at least two resolutions are four resolutions, and the highest resolution among the at least two resolutions is equal to the resolution of the image to be processed.

13. The method according to any one of claims 1-9, characterized in that, The first preset process includes at least one of the following: Image cutout processing; Background replacement processing.

14. The method according to any one of claims 1-9, characterized in that, The method further includes: Based on the processing result of the first preset processing, the image to be processed is subjected to a second preset processing, wherein the second preset processing includes at least one of the following: classification processing, retrieval processing, and display processing.

15. An image processing apparatus, characterized in that, include: The first image feature determination module is used to determine the first image features corresponding to the image to be processed; The first image feature processing module is used to determine the region classification information and image detail information corresponding to the image to be processed based on the first image features. The association processing module is used to perform association processing based on the region classification information and the image detail information to obtain context information; The fusion processing module is used to perform fusion processing based on the context information, the region classification information, and the image detail information to obtain enhanced features; The first preset processing module is used to determine the target region in the image to be processed based on the enhanced features, and to perform a first preset processing on the target region in the image to be processed.

16. The apparatus according to claim 15, characterized in that, The first image feature includes N third image features, where N is an integer greater than 1, and the N third image features correspond to at least two resolutions; The step of determining the region classification information and image detail information corresponding to the image to be processed based on the first image features includes: The region classification information is determined based on the third image features corresponding to the lowest resolution among the at least two resolutions; The image detail information is determined based on the third image feature corresponding to the highest of the at least two resolutions.

17. The apparatus according to claim 15, characterized in that, The process of associating the region classification information and the image detail information to obtain context information includes: Extract the information features of the region classification information and the image detail information respectively; An attention mechanism is used to correlate the extracted information features to obtain the context information.

18. The apparatus according to claim 15, characterized in that, The enhanced features are obtained by fusing the context information, the region classification information, and the image detail information, including: Based on each of the context information, the region classification information, and the image detail information, a dynamic weight model is used to generate weight information for each of them. Based on the weight information, the context information, the region classification information, and the image detail information are weighted and fused to obtain the enhanced features.

19. The apparatus according to claim 15, characterized in that, The determination of the first image feature corresponding to the image to be processed includes: The image to be processed is input into the encoder to obtain N second image features output by the encoder, wherein the N second image features correspond to at least two resolutions; Based on the N second image features, N third image features contained in the first image features are determined.

20. The apparatus according to any one of claims 15-19, characterized in that, The first preset process includes at least one of the following: Image cutout processing; Background replacement processing.

21. The apparatus according to any one of claims 15-19, characterized in that, The device further includes: The second preset processing module is used to perform a second preset processing on the image to be processed based on the processing result of the first preset processing, wherein the second preset processing includes at least one of the following: classification processing, retrieval processing, and display processing.

22. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-14.

23. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-14.

24. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-14.

Images