Image processing method, device, computer equipment and storage medium

Abstract

The present disclosure provides an image processing method, apparatus, computer device and storage medium, wherein the method comprises: obtaining an image to be processed; performing multiple rounds of visual marker generation processing and multiple rounds of serialized feature fusion processing on the image to be processed using a pre-trained feature extraction model to obtain a target feature map corresponding to the image to be processed.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the rapid development of neural network technology, an increasing number of network models capable of processing image data have emerged. The Transformer model, as a network model capable of processing image data, divides an image into rectangular regions of fixed shape and size, assigns a visual marker to each rectangular region, and then processes the sequenced image data using a sequence of visual markers. However, this approach needs to improve its accuracy in expressing image features and its ability to capture detailed information within the image. Summary of the Invention

[0003] The embodiments of the present disclosure at least provide an image processing method, apparatus, computer equipment, and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides an image processing method, comprising:

[0005] Get the image to be processed;

[0006] Performing multiple rounds of visual mark generation processing and multiple rounds of serialized feature fusion processing on the image to be processed using a pre-trained feature extraction model to obtain a target feature map corresponding to the image to be processed;

[0007] The n+1th round of visual mark generation processing includes the following steps:

[0008] Obtaining a first visual marker sequence output by the feature extraction model in an n-th round of visual marker generation processing, where visual markers in the first visual marker sequence represent image features of a portion of image content in the image to be processed;

[0009] performing clustering processing on each visual marker included in the first visual marker sequence based on a value of each visual marker included in the first visual marker sequence to obtain a target clustering result;

[0010] For each cluster in the target clustering result, the values ​​of the visual markers contained in the cluster are fused to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster;

[0011] Determining a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual markers corresponding to the clusters in the target clustering result;

[0012] The first visual marker sequence used in the first round of visual marker generation processing is generated based on an initial feature map obtained after feature extraction of the image to be processed, where n=1, 2, 3, ..., m, where m is a preset positive integer.

[0013] In a possible implementation, clustering the visual markers included in the first visual marker sequence based on the values ​​of the visual markers included in the first visual marker sequence to obtain a target clustering result includes:

[0014] splitting the first visual marker sequence based on a spatial relationship between the visual markers in the first visual marker sequence to obtain a plurality of visual marker blocks, each visual marker block including at least one visual marker;

[0015] For each visual marker block, clustering processing is performed on the visual markers included in the visual marker block based on the values ​​of the visual markers included in the visual marker block to obtain a clustering result corresponding to the visual marker block;

[0016] Based on the clustering results corresponding to each visual marker block, a target clustering result corresponding to the first visual marker sequence is generated.

[0017] In a possible implementation, the first visual marker sequence is split based on the spatial relationship between the visual markers in the first visual marker sequence to obtain multiple visual marker blocks, including:

[0018] Based on the preset number of blocks that matches the number of rounds of visual marker generation processing, and the spatial relationship between the visual markers in the first visual marker sequence, the first visual marker sequence is split to obtain a plurality of visual marker blocks whose number is the preset number of blocks; wherein the preset number of blocks corresponding to each round of visual marker generation processing decreases from round to round.

[0019] In this way, by using a decreasing number of blocks to perform block processing on the first visual marker sequence, the processing speed of the subsequent clustering processing can be improved by block processing during shallow feature extraction (that is, visual marker generation processing in the early rounds); on the other hand, when entering deep feature extraction (that is, visual marker generation processing in the later rounds), the impact of block processing on feature accuracy can be reduced round by round, thereby ensuring the accuracy when extracting deep features of the image and ensuring the accuracy of the target feature map finally obtained.

[0020] In one possible implementation, for each cluster in the target clustering result, fusing the values ​​of the visual markers contained in the cluster to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster includes:

[0021] For each cluster in the target clustering result, fusion is performed based on the values ​​corresponding to each visual marker in the cluster and the weight values ​​corresponding to each visual marker, so as to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster; wherein, the weight values ​​corresponding to each visual marker are determined based on the values ​​corresponding to each visual marker.

[0022] In this way, fusion can be performed according to the weight values ​​corresponding to each visual marker, so that the image features obtained after fusion can match the image content of each part of the image, so that the values ​​of the fused visual markers can better represent the image content.

[0023] In one possible implementation, determining the second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual marker corresponding to each cluster in the target clustering result includes:

[0024] Based on the spatial relationship between the visual markers corresponding to the clusters in the target clustering result, the visual markers corresponding to the clusters in the target clustering result are spliced ​​to obtain a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing.

[0025] In one possible implementation, the multiple rounds of serialized feature fusion processing include:

[0026] For the feature fusion processing of the nth round of serialization, based on the second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, a target feature map corresponding to the feature fusion processing of the nth round of serialization is generated;

[0027] Among them, the target feature map corresponding to the last round of serialized feature fusion processing is determined based on the second visual mark sequence corresponding to the last round of mark generation processing.

[0028] In one possible implementation, generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing includes:

[0029] Performing an upsampling process on the second visual marker sequence corresponding to the (n+1) round of visual marker generation processing to obtain an upsampled second visual marker sequence; wherein the upsampling parameters used in the upsampling process corresponding to the (n+1) round of visual marker generation processing are determined based on the target clustering result corresponding to the (n+1) round of visual marker generation processing;

[0030] Based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, a target feature map corresponding to the nth round of serialized feature fusion processing is generated.

[0031] In one possible implementation, generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing includes:

[0032] The second visual marker sequence after upsampling corresponding to the (n+1)th round of visual marker generation processing and the first visual marker sequence after linear transformation corresponding to the (n)th round of visual marker generation processing are fused to obtain a to-be-processed visual marker sequence corresponding to the (n)th round of serialized feature fusion processing;

[0033] Inputting the visual marker sequence to be processed into the serialization data processing module in the feature extraction model to obtain the target visual marker sequence corresponding to the feature fusion processing of the nth round of serialization;

[0034] The target visual mark sequence corresponding to the n-th round of serialized feature fusion processing is converted to obtain a converted target feature map corresponding to the n-th round of serialized feature fusion processing.

[0035] In a possible implementation manner, the method further includes:

[0036] Inputting the target feature map into a feature processing model to obtain an image processing result corresponding to the image to be processed;

[0037] The image processing result includes any one of the following results:

[0038] Semantic segmentation results, object recognition results, face recognition results, and stylization processing results.

[0039] In a second aspect, an embodiment of the present disclosure further provides an image processing device, including:

[0040] An acquisition module, used for acquiring an image to be processed;

[0041] A processing module is used to perform multiple rounds of visual mark generation processing and multiple rounds of serialized feature fusion processing on the image to be processed using a pre-trained feature extraction model to obtain a target feature map corresponding to the image to be processed;

[0042] The processing module, during the (n+1) round of visual mark generation processing, is used to:

[0043] Obtaining a first visual marker sequence output by the feature extraction model in an n-th round of visual marker generation processing, where visual markers in the first visual marker sequence represent image features of a portion of image content in the image to be processed;

[0044] performing clustering processing on each visual marker included in the first visual marker sequence based on a value of each visual marker included in the first visual marker sequence to obtain a target clustering result;

[0045] For each cluster in the target clustering result, the values ​​of the visual markers contained in the cluster are fused to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster;

[0046] Determining a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual markers corresponding to the clusters in the target clustering result;

[0047] The first visual marker sequence used in the first round of visual marker generation processing is generated based on an initial feature map obtained after feature extraction of the image to be processed, where n=1, 2, 3, ..., m, where m is a preset positive integer.

[0048] In one possible implementation, the processing module, when performing clustering processing on each visual marker included in the first visual marker sequence based on the value of each visual marker included in the first visual marker sequence to obtain a target clustering result, is configured to:

[0049] splitting the first visual marker sequence based on a spatial relationship between the visual markers in the first visual marker sequence to obtain a plurality of visual marker blocks, each visual marker block including at least one visual marker;

[0050] For any visual marker block, clustering is performed on the visual markers included in the visual marker block based on the values ​​of the visual markers included in the visual marker block to obtain a clustering result corresponding to the visual marker block;

[0051] Based on the clustering results corresponding to each visual marker block, a target clustering result corresponding to the first visual marker sequence is generated.

[0052] In one possible implementation, the processing module, when splitting the first visual marker sequence based on the spatial relationship between the visual markers in the first visual marker sequence to obtain a plurality of visual marker blocks, is configured to:

[0053] Based on the preset number of blocks that matches the number of rounds of visual marker generation processing, and the spatial relationship between the visual markers in the first visual marker sequence, the first visual marker sequence is split to obtain a plurality of visual marker blocks whose number is the preset number of blocks; wherein the preset number of blocks corresponding to each round of visual marker generation processing decreases from round to round.

[0054] In one possible implementation, the processing module, for each cluster in the target clustering result, fuses the values ​​of the visual markers contained in the cluster to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster, and is used to:

[0055] For each cluster in the target clustering result, fusion is performed based on the values ​​corresponding to each visual marker in the cluster and the weight values ​​corresponding to each visual marker, so as to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster; wherein, the weight values ​​corresponding to each visual marker are determined based on the values ​​corresponding to each visual marker.

[0056] In one possible implementation, the processing module, when determining the second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual markers corresponding to the clusters in the target clustering result, is configured to:

[0057] Based on the spatial relationship between the visual markers corresponding to the clusters in the target clustering result, the visual markers corresponding to the clusters in the target clustering result are spliced ​​to obtain a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing.

[0058] In one possible implementation, the processing module, during multiple rounds of serialized feature fusion processing, is configured to:

[0059] For the feature fusion processing of the nth round of serialization, based on the second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, a target feature map corresponding to the feature fusion processing of the nth round of serialization is generated;

[0060] Among them, the target feature map corresponding to the last round of serialized feature fusion processing is determined based on the second visual mark sequence corresponding to the last round of mark generation processing.

[0061] In one possible implementation, the processing module, when generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing, is configured to:

[0062] Performing an upsampling process on the second visual marker sequence corresponding to the (n+1) round of visual marker generation processing to obtain an upsampled second visual marker sequence; wherein the upsampling parameters used in the upsampling process corresponding to the (n+1) round of visual marker generation processing are determined based on the target clustering result corresponding to the (n+1) round of visual marker generation processing;

[0063] Based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, a target feature map corresponding to the nth round of serialized feature fusion processing is generated.

[0064] In one possible implementation, the processing module, when generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing, is configured to:

[0065] The second visual marker sequence after upsampling corresponding to the (n+1)th round of visual marker generation processing and the first visual marker sequence after linear transformation corresponding to the (n)th round of visual marker generation processing are fused to obtain a to-be-processed visual marker sequence corresponding to the (n)th round of serialized feature fusion processing;

[0066] Inputting the visual marker sequence to be processed into the serialization data processing module in the feature extraction model to obtain the target visual marker sequence corresponding to the feature fusion processing of the nth round of serialization;

[0067] The target visual mark sequence corresponding to the n-th round of serialized feature fusion processing is converted to obtain a converted target feature map corresponding to the n-th round of serialized feature fusion processing.

[0068] In a possible implementation manner, the processing module is further configured to:

[0069] Inputting the target feature map into a feature processing model to obtain an image processing result corresponding to the image to be processed;

[0070] The image processing result includes any one of the following results:

[0071] Semantic segmentation results, object recognition results, face recognition results, and stylization processing results.

[0072] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0073] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0074] The image processing method, apparatus, computer equipment and storage medium provided by the embodiments of the present disclosure can, during the process of performing multiple rounds of visual marker generation processing on the image to be processed, perform clustering processing based on the values ​​of each visual marker in the visual marker sequence obtained in the previous round, and fuse the visual markers based on the clustering results. In this way, the specific resolution of the feature map corresponding to the visual marker sequence obtained after multiple rounds of visual marker generation processing is related to the specific content of the image, so that features that can better express the image content and better capture detail information can be extracted; on the other hand, data processing based on the visual marker sequence after clustering and fusion can reduce computational complexity and thus improve processing speed.

[0075] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0077] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown;

[0078] Figure 2 A schematic diagram of generating a target feature map in the image processing method provided by an embodiment of the present disclosure is shown;

[0079] Figure 3A schematic diagram showing the processing process of the Transformer module in the image processing method provided by an embodiment of the present disclosure;

[0080] Figure 4 A flowchart of a visual marker generation process in an image processing method provided by an embodiment of the present disclosure is shown;

[0081] Figure 5 A schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure is shown;

[0082] Figure 6 A schematic structural diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0084] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0085] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0086] Research has found that in order to obtain the image features of the image to be processed at different scales, the Transformer model can perform multiple rounds of serialized image data processing when performing image processing, and in each round of image data processing, the adjacent visual markers at corresponding positions in the feature map are fused according to the relative position relationship of the values ​​of each visual marker in the visual marker sequence in the corresponding feature map, so as to obtain the image features of the image to be processed at different scales. However, the image features obtained in this way are only related to the rectangular area of ​​fixed shape and size used in the fusion processing of the image to be processed. Therefore, although this method can obtain the features of the image to be processed at different scales, it is easy to ignore the details in the image to be processed, so that the image feature extraction effect needs to be improved.

[0087] According to the relative position relationship of the values ​​of each visual marker in the corresponding feature map in the visual marker sequence, the adjacent visual markers at the corresponding positions in the feature map are fused to obtain the image features of the image to be processed at different scales. However, the image features obtained in this way are only related to the rectangular area of ​​fixed shape and size used in the fusion processing of the image to be processed. Therefore, although this method can obtain the features of the image to be processed at different scales, it is easy to ignore the details in the image to be processed, so that the image feature extraction effect needs to be improved.

[0088] Based on the above research, the present disclosure provides an image processing method, apparatus, computer equipment and storage medium. In the process of performing multiple rounds of visual marker generation processing on the image to be processed, clustering processing can be performed according to the values ​​of each visual marker in the visual marker sequence obtained in the previous round, and the visual markers can be fused according to the clustering results. In this way, the specific resolution of the feature map corresponding to the visual marker sequence obtained after multiple rounds of visual marker generation processing is related to the specific content of the image, so that the feature extraction model can better extract image features and capture detail information, thereby improving the performance of the feature extraction model in various image processing tasks; on the other hand, data processing according to the visual marker sequence after clustering and fusion can reduce the computational complexity, thereby improving the processing speed of the feature extraction model.

[0089] To facilitate understanding of this embodiment, an image processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the image processing method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities. The computer device includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the image processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0090] See also Figure 1 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure, wherein the method includes S101 to S102, wherein:

[0091] S101: Acquire an image to be processed.

[0092] S102: performing multiple rounds of visual tag generation processing and multiple rounds of serialized feature fusion processing on the image to be processed using a pre-trained feature extraction model to obtain a target feature map corresponding to the image to be processed.

[0093] The (n+1) round of visual marker generation processing includes the following steps S1021 to S1024:

[0094] S1021: Obtain a first visual marker sequence output by the feature extraction model in the nth round of visual marker generation processing, where the visual markers in the first visual marker sequence represent image features of part of the image content in the image to be processed.

[0095] S1022: Based on the value of each visual marker included in the first visual marker sequence, cluster the visual markers included in the first visual marker sequence to obtain a target clustering result.

[0096] S1023: For each cluster in the target clustering result, the values ​​of the visual markers contained in the cluster are fused to obtain a visual marker fusion result of each cluster as the visual marker corresponding to the cluster.

[0097] S1024: Determine a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual marker corresponding to each cluster in the target clustering result.

[0098] Among them, the first visual marker sequence used in the first round of visual marker generation processing is generated based on the initial feature map obtained after feature extraction of the image to be processed, n=1, 2, 3, ..., m, where m is a preset positive integer.

[0099] The following is a detailed description of the above steps.

[0100] The image to be processed may be an image corresponding to a target processing requirement, and the target processing requirement may include image classification, key point detection, image segmentation, object detection, face recognition, etc.; the network type of the feature extraction model may be a neural network model with serialized image data processing capabilities, such as a Transformer model.

[0101] Specifically, after obtaining the image to be processed, if it is detected that the image size of the image to be processed does not meet the input size requirement of the feature extraction model, the size of the image to be processed can be adjusted, and the image to be processed after the size adjustment can be used to perform subsequent processing steps.

[0102] The following will introduce the specific processing of visual marker generation processing and serialized feature fusion processing respectively. First, the specific processing of visual marker generation processing is introduced:

[0103] For S1021,

[0104] Here, during the n+1th round of visual marker generation processing, the first visual marker sequence output by the feature extraction model in the nth round of visual marker generation processing is obtained, that is, during this round of visual marker generation processing, the visual marker sequence obtained by the previous round of visual marker generation processing is first obtained; and the first visual marker sequence used in the first round of visual marker generation processing is generated based on the initial feature map obtained after feature extraction of the image to be processed.

[0105] Specifically, when generating the first visual marker sequence used in the first round of visual marker generation processing, feature extraction can be performed on the image to be processed to obtain an initial feature map corresponding to the image to be processed; wherein, when feature extraction is performed on the image to be processed, feature extraction can be performed on the image to be processed through the convolutional layer in the feature extraction model to obtain an initial feature map corresponding to the image to be processed.

[0106] Furthermore, after obtaining the initial feature map, the initial feature values ​​can be converted into a first visual mark sequence used in the first round of visual mark generation processing according to the positions of the eigenvalues ​​(eigenvalues ​​can include values ​​of multiple channels, and eigenvalues ​​are also eigenvectors) in the initial feature map; a eigenvalue in the initial feature map corresponds to a visual mark in the first visual mark sequence, that is, the visual mark in the visual mark sequence corresponds to the eigenvalue in the feature map, so the visual mark can represent the image characteristics of part of the image content in the image to be processed.

[0107] For S1022,

[0108] Here, clustering is performed on the visual markers included in the first visual marker sequence to obtain a target clustering result containing multiple clusters, so that the visual markers in the first visual marker sequence can be fused through subsequent steps to achieve downsampling of the visual markers, so that deeper image features can be gradually obtained during multiple rounds of processing, so as to obtain more accurate image processing results through image features at different levels; when clustering, clustering can be performed through a preset clustering algorithm, and the clustering algorithm can be, for example, a K-Nearest Neighbors (KNN) clustering algorithm, a Mean-Shift clustering algorithm based on a sliding window, a density-based noise application space clustering algorithm, etc. The embodiment of the present disclosure does not limit the clustering algorithm to be used, and is subject to what can be implemented.

[0109] In one possible implementation, when clustering the visual markers included in the first visual marker sequence based on the values ​​of the visual markers included in the first visual marker sequence to obtain a target clustering result, the following steps A1 to A3 may be performed:

[0110] A1: Based on the spatial relationship between the visual markers in the first visual marker sequence, the first visual marker sequence is split to obtain a plurality of visual marker blocks, each visual marker block including at least one visual marker.

[0111] Here, the spatial relationship between the visual markers can be the spatial relationship between the feature values ​​in the feature map corresponding to each visual marker. For example, if they are located in the upper left, upper right, lower left, and lower right of the feature map respectively, the first visual marker sequence can be split into 4 visual marker blocks according to the spatial relationship. The 4 visual marker blocks correspond to the upper left area, upper right area, lower left area, and lower right area of ​​the feature map respectively; the visual markers contained in each visual marker block can form a visual marker subsequence, so that the subsequent serialized data processing process can be continued according to the formed visual marker subsequence.

[0112] In one possible embodiment, when the first visual marker sequence is split based on the spatial relationship between each visual marker in the first visual marker sequence to obtain a plurality of visual marker blocks, the first visual marker sequence can be split based on a preset number of blocks that matches the number of rounds of visual marker generation processing, and the spatial relationship between each visual marker in the first visual marker sequence to obtain a plurality of visual marker blocks whose number is the preset number of blocks.

[0113] Among them, the number of preset blocks corresponding to each round of visual mark generation processing decreases from round to round.

[0114] For example, taking the case where the number of rounds of visual marker generation processing is 3, the preset number of blocks corresponding to each round can be 16, 4, and 1 (the number of blocks is 1, which means no block processing is performed). In this way, the processing efficiency and feature accuracy can be taken into account by decreasing the number of blocks in each round.

[0115] In this way, by using a decreasing number of blocks to perform block processing on the first visual marker sequence, the processing speed of the subsequent clustering processing can be improved by block processing during shallow feature extraction (that is, visual marker generation processing in the early rounds); on the other hand, when entering deep feature extraction (that is, visual marker generation processing in the later rounds), the impact of block processing on feature accuracy can be reduced round by round, that is, the impact of blocking on accuracy when extracting deep features of the image can be reduced, thereby ensuring the accuracy of the target feature map finally obtained.

[0116] A2: For each visual marker block, cluster the visual markers included in the visual marker block based on the values ​​of the visual markers included in the visual marker block to obtain a clustering result corresponding to the visual marker block.

[0117] A3: Based on the clustering results corresponding to each visual marker block, generate a target clustering result corresponding to the first visual marker sequence.

[0118] For example, when performing clustering processing, the following formula may be used:

[0119]

[0120]

[0121] Wherein, i represents the serial number corresponding to the visual marker in a visual marker block, j represents the serial number corresponding to other visual markers other than the serial number i in a visual marker block; represents the distribution density of the cluster center corresponding to the i-th visual marker in the estimated visual marker block, which is used to characterize the distribution of the cluster centers in the visual marker block; represents the value of the visual marker with the serial number i in the visual marker block (the value of multiple channels, that is, a vector), and k represents the number of visual markers; the above formula (1) is used to calculate the distribution density of the cluster center corresponding to each point in the visual marker block according to the distance between each visual marker and other visual markers in the same visual marker block;

[0122] It represents the calculation rule when calculating the distance between the i-th visual marker in the visual marker block and other visual markers in the visual marker block when the visual marker is used as the cluster center. Therefore, the product of the sum can represent the cluster score of the i-th visual marker in the visual marker block becoming the cluster center. The higher the cluster score, the greater the probability that it is the cluster center. Formula (2) is used to judge whether there is a visual marker j with a larger distribution density than the current visual marker i, and different distance calculation methods are used according to the judgment result. If there is a visual marker j with a larger distribution density than the current visual marker i, the first row of the above formula (2) is used for calculation to obtain the distance between the visual marker i and the current j. If there is no visual marker j with a larger distribution density than the current visual marker i, the second row of the above formula (2) is used for calculation to obtain the distance between the visual marker i and the current j.

[0123] Specifically, after the sum is calculated according to the above formula, the corresponding δ in each visual marker block can be i The visual marker with the largest product of ρi and ρi is used as the cluster center of each visual marker block, thereby obtaining the clustering results corresponding to each visual marker block. The clustering results corresponding to each visual marker block constitute the target clustering results corresponding to the first visual marker sequence.

[0124] In this way, by clustering each visual marker block after dividing the visual marker sequence into blocks, the number of visual markers can be further reduced in combination with subsequent fusion steps to achieve gradual fusion and processing of image features, thereby extracting deeper features of the image.

[0125] For S1023,

[0126] Here, for each cluster in the target clustering result, the values ​​of the visual markers contained in the cluster can be fused to obtain the fused visual marker corresponding to the cluster, thereby obtaining the visual marker fusion result of each cluster as the visual marker corresponding to the cluster, and achieving feature fusion by reducing the number of visual markers.

[0127] Specifically, when the values ​​of the visual markers contained in the cluster are fused, the values ​​of the visual markers contained in the cluster can be averaged, and the average value of the obtained values ​​is used as the value of the fused visual marker, thereby realizing the fusion of the visual markers.

[0128] In one possible implementation, for each cluster in the target clustering results, when the values ​​of the visual markers contained in the cluster are fused to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster, for each cluster in the target clustering results, the values ​​corresponding to each visual marker in the cluster and the weight values ​​corresponding to each visual marker can be fused to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster.

[0129] The weight values ​​corresponding to the visual markers are determined based on the values ​​corresponding to the visual markers.

[0130] Specifically, when determining the weight values ​​corresponding to each visual marker, the weights corresponding to each visual marker can be predicted according to the values ​​corresponding to each visual marker through the fully connected layer, thereby achieving weighted averaging of the visual markers within each cluster.

[0131] For example, when performing weighted averaging on the visual markers within each cluster, the following formula can be used:

[0132]

[0133] Among them, y i Indicates the calculation result obtained after calculation; C i represents the visual marker block corresponding to the i-th cluster, x j represents the value corresponding to the jth visual mark, p j Represents the weight value corresponding to the j-th visual marker.

[0134] In this way, by performing weighted averaging processing on the visual markers, they can be fused according to the weight values ​​corresponding to each visual marker, so that the image features obtained after fusion match the image content of each part of the image, and the values ​​of the fused visual markers can better represent the image content.

[0135] For S1024,

[0136] In one possible implementation, when determining the second visual marker sequence corresponding to the n+1th round of visual marker generation processing based on the visual markers corresponding to each cluster in the target clustering result, the visual markers corresponding to each cluster in the target clustering result can be spliced ​​based on the spatial relationship between the visual markers corresponding to each cluster in the target clustering result to obtain the second visual marker sequence corresponding to the n+1th round of visual marker generation processing.

[0137] Here, the spatial relationship between the visual markers corresponding to each cluster in the target clustering result can be the spatial relationship between the eigenvalues ​​in the feature map corresponding to the visual markers corresponding to each cluster. Based on the spatial relationship between the visual markers corresponding to each cluster in the target clustering result, the visual markers corresponding to each cluster in the target clustering result are spliced ​​to obtain a second visual marker sequence that matches the spatial relationship of the corresponding eigenvalues ​​in the feature map, so that the feature map matching the image to be processed can be obtained by converting the visual markers in the second visual marker sequence.

[0138] In one possible implementation, during multiple rounds of serialized feature fusion processing, for the nth round of serialized feature fusion processing, a target feature map corresponding to the nth round of serialized feature fusion processing is generated based on the second visual marker sequence corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing.

[0139] Among them, the target feature map corresponding to the last round of serialized feature fusion processing is determined based on the second visual mark sequence corresponding to the last round of mark generation processing.

[0140] For example, taking the number of rounds m as 3 as an example, the schematic diagram of generating the target feature map can be as follows: Figure 2 As shown, Figure 2 In the figure, the corresponding visual marker sequences (i.e., the second visual marker sequence) are obtained after the 1st, 2nd, and 3rd rounds of visual marker generation processing respectively. The target feature map corresponding to the last round (i.e., the 3rd round) of feature fusion processing is obtained by taking the visual marker sequence obtained after the 3rd round of visual marker generation processing as the target visual marker sequence and performing conversion processing; the target feature map corresponding to the 2nd round of feature fusion processing is determined based on the visual marker sequence output by the 3rd round of visual marker generation processing and the visual marker sequence output by the 2nd round of visual marker generation processing; the target feature map corresponding to the 1st round of feature fusion processing is determined based on the visual marker sequence output by the 2nd round of visual marker generation processing and the visual marker sequence output by the 1st round of visual marker generation processing.

[0141] In one possible implementation, when generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing, the following steps B1 to B2 may be performed:

[0142] B1: Upsampling the second visual marker sequence corresponding to the n+1th round of visual marker generation processing to obtain a second visual marker sequence after upsampling processing; wherein, the upsampling parameters used in the upsampling processing corresponding to the n+1th round of visual marker generation processing are determined based on the target clustering results corresponding to the n+1th round of visual marker generation processing.

[0143] Here, since it is necessary to fuse the second visual marker sequence corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, it is necessary to upsample the second visual marker sequence corresponding to the n+1th round of visual marker generation processing according to the corresponding upsampling parameters, so as to obtain the upsampled second visual marker sequence that can be fused with the first visual marker sequence output by the nth round of visual marker generation processing.

[0144] B2: Based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, generate the target feature map corresponding to the nth round of serialized feature fusion processing.

[0145] In one possible implementation, when generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing, the following steps B21 to B23 may be performed:

[0146] B21: Fuse the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence after linear transformation corresponding to the nth round of visual marker generation processing to obtain the visual marker sequence to be processed corresponding to the nth round of serialized feature fusion processing.

[0147] B22: Input the visual marker sequence to be processed into the serialization data processing module in the feature extraction model to obtain the target visual marker sequence corresponding to the feature fusion processing of the nth round of serialization.

[0148] Here, the serialized data processing module may be a Transformer module.

[0149] For example, the processing of the Transformer module can be as follows Figure 3 As shown, Figure 3 In the method, the input visual marker sequence to be processed can be input into the normalization layer, and then down-sampled in the weight matrix V and the weight matrix K, and input into the multi-head attention layer together with the down-sampled weight matrix Q for processing. Then, the processing results of the multi-head attention layer and the visual marker sequence to be processed are fused, and the fused results are sequentially input into the normalization layer, the linearization processing layer, the convolution layer, the activation layer and the linearization processing layer for processing, and fusion processing is performed according to the processing results and the fused results to obtain the final target visual marker sequence.

[0150] B23: Convert the target visual mark sequence corresponding to the n-th round of serialized feature fusion processing to obtain a converted target feature map corresponding to the n-th round of serialized feature fusion processing.

[0151] Here, by converting the target visual mark sequence corresponding to the feature fusion processing of the nth round of serialization, a converted target feature map corresponding to the feature fusion processing of the nth round of serialization is obtained.

[0152] Furthermore, after obtaining the target feature map, the target feature map can be input into a feature processing model to obtain an image processing result corresponding to the image to be processed;

[0153] The image processing result includes any one of the following results:

[0154] Semantic segmentation results, object recognition results, face recognition results, and stylization processing results.

[0155] Specifically, the feature processing model can be a pre-trained neural network model, and the network type of the feature processing model can be a convolutional neural network (CNN), a deep residual network (ResNet), a recurrent neural network (RNN), a Transformer model, etc.

[0156] Below, combined with the attached Figure 4 The specific processing flow of the visual marker generation process in the image processing method provided by the embodiment of the present disclosure is introduced. Figure 4 The number of rounds of visual marker generation processing included in is 3, including the following steps:

[0157] 1. Input the original image to be processed into the convolution layer to obtain the initial feature map.

[0158] 2. Perform visual marker generation processing based on the initial feature map to generate a visual marker sequence.

[0159] At this time, the visual marker sequence has not undergone clustering and fusion processing. The eigenvalues ​​in the feature map corresponding to each visual marker (that is, the image containing multiple grids in the map) are all part of the feature map, and the sizes of the corresponding areas are the same (grids filled with different colors in the map correspond to a visual marker, and the size of the grid corresponding to each visual marker is the same).

[0160] Specifically, the process of generating the corresponding target feature map according to the visual mark sequence can refer to the above Figure 2 and Figure 3 The description of the corresponding content will not be repeated here.

[0161] 3. Perform clustering and fusion processing on the visual marker sequence to dynamically cluster the visual marker sequence according to the value of the visual marker, so as to generate a new visual marker sequence.

[0162] At this time, the visual marker sequence has been clustered and fused, and the eigenvalues ​​in the feature map corresponding to the visual marker are all part of the feature map area, but the sizes of the corresponding areas can be different (that is, the sizes of the grids in the feature map corresponding to the visual marker can be different).

[0163] 4. Perform feature fusion processing on the visual marker sequence obtained from each round of visual marker generation processing to obtain the target feature map corresponding to the original image.

[0164] Specifically, the specific contents of the above steps can be referred to the relevant contents above and will not be repeated here.

[0165] The image processing method provided by the embodiment of the present disclosure can, in the process of performing multiple rounds of visual marker generation processing on the image to be processed, perform clustering processing based on the values ​​of each visual marker in the visual marker sequence obtained in the previous round, and fuse the visual markers based on the clustering results. In this way, the specific resolution of the feature map corresponding to the visual marker sequence obtained after multiple rounds of visual marker generation processing is related to the specific content of the image, so that features that can better express the image content and better capture detail information can be extracted; on the other hand, data processing based on the visual marker sequence after clustering and fusion can reduce computational complexity and thus improve processing speed.

[0166] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0167] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0168] Reference Figure 5 FIG. 5 is a schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure, wherein the device includes: an acquisition module 501 and a processing module 502; wherein,

[0169] An acquisition module 501 is used to acquire an image to be processed;

[0170] A processing module 502 is configured to perform multiple rounds of visual tag generation processing and multiple rounds of serialized feature fusion processing on the image to be processed using a pre-trained feature extraction model to obtain a target feature map corresponding to the image to be processed;

[0171] The processing module 502 is used, during the (n+1)th round of visual mark generation processing, to:

[0172] Obtaining a first visual marker sequence output by the feature extraction model in an n-th round of visual marker generation processing, where visual markers in the first visual marker sequence represent image features of a portion of image content in the image to be processed;

[0173] performing clustering processing on each visual marker included in the first visual marker sequence based on a value of each visual marker included in the first visual marker sequence to obtain a target clustering result;

[0174] For each cluster in the target clustering result, the values ​​of the visual markers contained in the cluster are fused to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster;

[0175] Determining a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual markers corresponding to the clusters in the target clustering result;

[0176] The first visual marker sequence used in the first round of visual marker generation processing is generated based on an initial feature map obtained after feature extraction of the image to be processed, where n=1, 2, 3, ..., m, where m is a preset positive integer.

[0177] In one possible implementation, the processing module 502, when performing clustering processing on each visual marker included in the first visual marker sequence based on the value of each visual marker included in the first visual marker sequence to obtain a target clustering result, is configured to:

[0178] splitting the first visual marker sequence based on a spatial relationship between the visual markers in the first visual marker sequence to obtain a plurality of visual marker blocks, each visual marker block including at least one visual marker;

[0179] For each visual marker block, clustering processing is performed on the visual markers included in the visual marker block based on the values ​​of the visual markers included in the visual marker block to obtain a clustering result corresponding to the visual marker block;

[0180] Based on the clustering results corresponding to each visual marker block, a target clustering result corresponding to the first visual marker sequence is generated.

[0181] In one possible implementation, the processing module 502, when splitting the first visual marker sequence based on the spatial relationship between the visual markers in the first visual marker sequence to obtain a plurality of visual marker blocks, is configured to:

[0182] Based on the preset number of blocks that matches the number of rounds of visual marker generation processing, and the spatial relationship between the visual markers in the first visual marker sequence, the first visual marker sequence is split to obtain a plurality of visual marker blocks whose number is the preset number of blocks; wherein the preset number of blocks corresponding to each round of visual marker generation processing decreases from round to round.

[0183] In one possible implementation, the processing module 502, for each cluster in the target clustering result, fuses the values ​​of the visual markers contained in the cluster to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster, and is configured to:

[0184] For each cluster in the target clustering result, fusion is performed based on the values ​​corresponding to each visual marker in the cluster and the weight values ​​corresponding to each visual marker, so as to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster; wherein, the weight values ​​corresponding to each visual marker are determined based on the values ​​corresponding to each visual marker.

[0185] In one possible implementation, the processing module 502, when determining the second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual markers corresponding to the clusters in the target clustering result, is configured to:

[0186] Based on the spatial relationship between the visual markers corresponding to the clusters in the target clustering result, the visual markers corresponding to the clusters in the target clustering result are spliced ​​to obtain a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing.

[0187] In one possible implementation, the processing module 502, during multiple rounds of serialized feature fusion processing, is configured to:

[0188] For the feature fusion processing of the nth round of serialization, based on the second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, a target feature map corresponding to the feature fusion processing of the nth round of serialization is generated;

[0189] Among them, the target feature map corresponding to the last round of serialized feature fusion processing is determined based on the second visual mark sequence corresponding to the last round of mark generation processing.

[0190] In one possible implementation, the processing module 502, when generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing, is configured to:

[0191] Performing an upsampling process on the second visual marker sequence corresponding to the (n+1) round of visual marker generation processing to obtain an upsampled second visual marker sequence; wherein the upsampling parameters used in the upsampling process corresponding to the (n+1) round of visual marker generation processing are determined based on the target clustering result corresponding to the (n+1) round of visual marker generation processing;

[0192] Based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, a target feature map corresponding to the nth round of serialized feature fusion processing is generated.

[0193] In one possible implementation, the processing module 502, when generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted from the nth round of visual marker generation processing, is configured to:

[0194] The second visual marker sequence after upsampling corresponding to the (n+1)th round of visual marker generation processing and the first visual marker sequence after linear transformation corresponding to the (n)th round of visual marker generation processing are fused to obtain a to-be-processed visual marker sequence corresponding to the (n)th round of serialized feature fusion processing;

[0195] Input the visual marker sequence to be processed into the serialization data processing module 502 in the feature extraction model to obtain the target visual marker sequence corresponding to the feature fusion processing of the nth round of serialization;

[0196] The target visual mark sequence corresponding to the n-th round of serialized feature fusion processing is converted to obtain a converted target feature map corresponding to the n-th round of serialized feature fusion processing.

[0197] In a possible implementation, the processing module 502 is further configured to:

[0198] Inputting the target feature map into a feature processing model to obtain an image processing result corresponding to the image to be processed;

[0199] The image processing result includes any one of the following results:

[0200] Semantic segmentation results, object recognition results, face recognition results, and stylization processing results.

[0201] The image processing device provided by the embodiment of the present disclosure can, in the process of performing multiple rounds of visual marker generation processing on the image to be processed, perform clustering processing based on the values ​​of each visual marker in the visual marker sequence obtained in the previous round, and fuse the visual markers based on the clustering results. In this way, the specific resolution of the feature map corresponding to the visual marker sequence obtained after multiple rounds of visual marker generation processing is related to the specific content of the image, so that features that can better express the image content and better capture detail information can be extracted; on the other hand, data processing based on the visual marker sequence after clustering and fusion can reduce computational complexity and thus improve processing speed.

[0202] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0203] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 6 6 is a schematic diagram of the structure of a computer device 600 provided in an embodiment of the present disclosure, including a processor 601, a memory 602, and a bus 603. The memory 602 is used to store execution instructions and includes a memory 6021 and an external memory 6022. The memory 6021 is also referred to as internal memory and is used to temporarily store operation data in the processor 601 and data exchanged with an external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the memory 6021. When the computer device 600 is running, the processor 601 communicates with the memory 602 via the bus 603, so that the processor 601 executes the following instructions:

[0204] Get the image to be processed;

[0205] Performing multiple rounds of visual mark generation processing and multiple rounds of serialized feature fusion processing on the image to be processed using a pre-trained feature extraction model to obtain a target feature map corresponding to the image to be processed;

[0206] The n+1th round of visual mark generation processing includes the following steps:

[0207] Obtaining a first visual marker sequence output by the feature extraction model in an n-th round of visual marker generation processing, where visual markers in the first visual marker sequence represent image features of a portion of image content in the image to be processed;

[0208] performing clustering processing on each visual marker included in the first visual marker sequence based on a value of each visual marker included in the first visual marker sequence to obtain a target clustering result;

[0209] For each cluster in the target clustering result, the values ​​of the visual markers contained in the cluster are fused to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster;

[0210] Determining a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual markers corresponding to the clusters in the target clustering result;

[0211] The first visual marker sequence used in the first round of visual marker generation processing is generated based on an initial feature map obtained after feature extraction of the image to be processed, where n=1, 2, 3, ..., m, where m is a preset positive integer.

[0212] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the image processing method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0213] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0214] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0215] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0216] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0217] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0218] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0219] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Performing multiple rounds of visual mark generation processing and multiple rounds of serialized feature fusion processing on the image to be processed using a pre-trained feature extraction model to obtain a target feature map corresponding to the image to be processed; The n+1th round of visual mark generation processing includes the following steps: Obtaining a first visual marker sequence output by the feature extraction model in an n-th round of visual marker generation processing, where visual markers in the first visual marker sequence represent image features of a portion of image content in the image to be processed; performing clustering processing on each visual marker included in the first visual marker sequence based on a value of each visual marker included in the first visual marker sequence to obtain a target clustering result; For each cluster in the target clustering result, the values ​​of the visual markers contained in the cluster are fused to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster; Determining a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual markers corresponding to the clusters in the target clustering result; The first visual marker sequence used in the first round of visual marker generation processing is generated based on an initial feature map obtained after feature extraction of the image to be processed, where n=1, 2, 3, ..., m, where m is a preset positive integer.

2. The method according to claim 1, characterized in that The clustering process is performed on the visual markers included in the first visual marker sequence based on the values ​​of the visual markers included in the first visual marker sequence to obtain a target clustering result, including: splitting the first visual marker sequence based on a spatial relationship between the visual markers in the first visual marker sequence to obtain a plurality of visual marker blocks, each visual marker block including at least one visual marker; For each visual marker block, clustering processing is performed on the visual markers included in the visual marker block based on the values ​​of the visual markers included in the visual marker block to obtain a clustering result corresponding to the visual marker block; Based on the clustering results corresponding to each visual marker block, a target clustering result corresponding to the first visual marker sequence is generated.

3. The method according to claim 2, characterized in that The first visual marker sequence is split based on the spatial relationship between the visual markers in the first visual marker sequence to obtain a plurality of visual marker blocks, including: Based on the preset number of blocks that matches the number of rounds of visual marker generation processing, and the spatial relationship between the visual markers in the first visual marker sequence, the first visual marker sequence is split to obtain a plurality of visual marker blocks whose number is the preset number of blocks; wherein the preset number of blocks corresponding to each round of visual marker generation processing decreases from round to round.

4. The method according to claim 1, wherein For each cluster in the target clustering result, fusing the values ​​of the visual markers contained in the cluster to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster, including: For each cluster in the target clustering result, fusion is performed based on the values ​​corresponding to each visual marker in the cluster and the weight values ​​corresponding to each visual marker, so as to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster; wherein, the weight values ​​corresponding to each visual marker are determined based on the values ​​corresponding to each visual marker.

5. The method according to claim 1, characterized in that The determining, based on the visual markers corresponding to the clusters in the target clustering result, a second visual marker sequence corresponding to the (n+1) round of visual marker generation processing includes: Based on the spatial relationship between the visual markers corresponding to the clusters in the target clustering result, the visual markers corresponding to the clusters in the target clustering result are spliced ​​to obtain a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing.

6. The method according to claim 1, characterized in that The multi-round serialized feature fusion processing includes: For the feature fusion processing of the nth round of serialization, based on the second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, a target feature map corresponding to the feature fusion processing of the nth round of serialization is generated; Among them, the target feature map corresponding to the last round of serialized feature fusion processing is determined based on the second visual mark sequence corresponding to the last round of mark generation processing.

7. The method according to claim 6, characterized in that The step of generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing includes: Performing an upsampling process on the second visual marker sequence corresponding to the (n+1) round of visual marker generation processing to obtain an upsampled second visual marker sequence; wherein the upsampling parameters used in the upsampling process corresponding to the (n+1) round of visual marker generation processing are determined based on the target clustering result corresponding to the (n+1) round of visual marker generation processing; Based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence output by the nth round of visual marker generation processing, a target feature map corresponding to the nth round of serialized feature fusion processing is generated.

8. The method according to claim 7, characterized in that The method of generating a target feature map corresponding to the nth round of serialized feature fusion processing based on the second visual marker sequence after upsampling processing corresponding to the n+1th round of visual marker generation processing and the first visual marker sequence outputted by the nth round of visual marker generation processing includes: The second visual marker sequence after upsampling corresponding to the (n+1)th round of visual marker generation processing and the first visual marker sequence after linear transformation corresponding to the (n)th round of visual marker generation processing are fused to obtain a to-be-processed visual marker sequence corresponding to the (n)th round of serialized feature fusion processing; Inputting the visual marker sequence to be processed into the serialization data processing module in the feature extraction model to obtain the target visual marker sequence corresponding to the feature fusion processing of the nth round of serialization; The target visual mark sequence corresponding to the n-th round of serialized feature fusion processing is converted to obtain a converted target feature map corresponding to the n-th round of serialized feature fusion processing.

9. The method according to claim 1, characterized in that The method further comprises: Inputting the target feature map into a feature processing model to obtain an image processing result corresponding to the image to be processed; The image processing result includes any one of the following results: Semantic segmentation results, object recognition results, face recognition results, and stylization processing results.

10. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A processing module is used to perform multiple rounds of visual mark generation processing and multiple rounds of serialized feature fusion processing on the image to be processed using a pre-trained feature extraction model to obtain a target feature map corresponding to the image to be processed; The processing module, during the (n+1) round of visual mark generation processing, is used to: Obtaining a first visual marker sequence output by the feature extraction model in an n-th round of visual marker generation processing, where visual markers in the first visual marker sequence represent image features of a portion of image content in the image to be processed; performing clustering processing on each visual marker included in the first visual marker sequence based on a value of each visual marker included in the first visual marker sequence to obtain a target clustering result; For each cluster in the target clustering result, the values ​​of the visual markers contained in the cluster are fused to obtain the visual marker fusion result of each cluster as the visual marker corresponding to the cluster; Determining a second visual marker sequence corresponding to the (n+1)th round of visual marker generation processing based on the visual markers corresponding to the clusters in the target clustering result; The first visual marker sequence used in the first round of visual marker generation processing is generated based on an initial feature map obtained after feature extraction of the image to be processed, where n=1, 2, 3, ..., m, where m is a preset positive integer.

11. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 9 are performed.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the image processing method according to any one of claims 1 to 9.

Images