Superpixel semantic primitive generation method and system, electronic equipment and storage medium

Abstract

The invention discloses a superpixel semantic primitive generation method and system, electronic equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: constructing a superpixel semantic primitive generation model; training the super-pixel semantic primitive generation model to obtain a trained super-pixel semantic primitive generation model; processing a target image by using the trained super-pixel semantic primitive generation model to obtain super-pixel semantic primitive features of the target image; according to the superpixel semantic primitive features of the target image, establishing a soft association relationship between pixel points of the target image and superpixel semantic primitives of the target image; and obtaining the superpixel semantic primitive of the target image according to the pixel points of the target image and the soft association relationship. According to the method, the superpixel semantic primitives can be efficiently and accurately generated.

Description

Background Art

[0002] With the rapid development of the Internet and digital technology, multimodal information is increasing rapidly. Traditional information retrieval and processing methods based on a single modality (such as plain text) often seem to be unable to cope with multimodal documents. In order to effectively meet this challenge, the research on multimodal document information processing has emerged. Multimodal documents usually include multiple information sources such as text, images, and charts. These information sources jointly convey the core content of the document through contextual semantic connections. Images are not only a supplement to text information, but also bear the task of directly expressing the core of the document in many cases. Therefore, how to efficiently and accurately extract semantic primitives from images that can be used for document understanding has become the key to improving document information retrieval and automatic understanding capabilities. Superpixel segmentation technology, as an effective way to reduce the number of pixel-level primitives in images and divide images into pixel blocks rich in semantic information, can provide more angles for generating multimodal document image semantic primitives. Superpixel segmentation is a method of aggregating similar pixels into larger units, namely superpixels. The core advantage of this method is that it can reduce the complexity of image data while maintaining important edge and structural information of the image to a certain extent.

[0003] At present, many superpixel algorithms based on traditional image processing technology and machine learning technology have been proposed and widely used. However, most of these traditional algorithms are difficult to parallelize, which makes them difficult to integrate into the existing large-scale multimodal document processing framework. In recent years, some researchers have begun to use deep network learning technology to improve the performance of superpixel segmentation. The advantage of this type of method is that it can embed the superpixel segmentation process into a large-scale parallel processing framework, and generate superpixel semantic primitives of the image during segmentation, thereby providing more accurate semantic support for subsequent understanding and analysis. However, the existing models are difficult to balance in terms of computational efficiency, boundary adhesion, and robustness. Therefore, how to design a superpixel semantic primitive generation network based on deep learning to reduce the amount of superpixel segmentation calculations while improving the accuracy of segmentation and the quality of semantic primitive features is one of the difficulties currently faced in research.

[0004] In the generation of multimodal image semantic primitives based on superpixels, it is necessary not only to consider how to extract the features of each pixel, but also to consider how to use the extracted features to assign the pixels to the corresponding superpixels. For feature extraction, many current methods either directly use the initial features of the image for weighting or use convolutional neural networks for feature extraction, but ignore the role of edges. In fact, the edge is one of the most important features in an image, which depicts the outline and shape of the object. When performing semantic primitive generation, paying attention to the image boundary can ensure the integrity of the object in the image and avoid erroneously classifying pixels belonging to different objects into the same superpixel semantic primitive. As for how to assign pixels to the corresponding superpixel semantic primitives, there are currently methods based on graph partitioning, clustering, regional evolution, and deep learning. The first three methods have the problems of low computational efficiency and difficulty in parallel computing; the method based on deep learning is fast, but has high requirements on the quality of labeled data and poor generalization performance. Therefore, how to make full use of the edge features of image objects and how to accurately and stably generate superpixel semantic primitives are of great research significance, and at the same time, this also brings certain challenges to the further development of current multimodal document processing. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a superpixel semantic primitive generation method, system, electronic device and storage medium, as follows:

[0006] 1) In the first aspect, the present invention provides a method for generating superpixel semantic primitives, and the specific technical solution is as follows:

[0007] Construct a superpixel semantic primitive generation model;

[0008] Training the superpixel semantic primitive generation model to obtain a trained superpixel semantic primitive generation model;

[0009] The target image is processed using the trained super-pixel semantic primitive generation model to obtain the super-pixel semantic primitive features of the target image;

[0010] According to the super-pixel semantic primitive features of the target image, a soft correlation relationship between the pixel points of the target image and the super-pixel semantic primitives of the target image is established;

[0011] According to the pixel points and soft association relationship of the target image, the superpixel semantic primitive of the target image is obtained.

[0012] Based on the above solution, the superpixel semantic primitive generation method of the present invention can also be improved as follows.

[0013] Furthermore, the superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged in sequence, n decoders arranged in sequence and n-2 edge supplementation modules; the first encoder is used to receive a sample image, the n encoders and n decoders correspond one to one, and in two adjacent encoders, the output of the front encoder is used as the input of the rear encoder, and in two adjacent decoders, the output of the front decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder, and the output of the second encoder to the n-1th encoder is respectively processed by an edge supplementation module and then input into the corresponding decoder; the output of the nth encoder and the output of the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

[0014] Furthermore, the data processing process of the edge supplement module is:

[0015] The received data is processed by a first convolution layer and a second convolution layer respectively to obtain a first eigenvector and a second eigenvector;

[0016] After performing the channel attention operation on the first eigenvector, multiply it with the first eigenvector to obtain the third eigenvector;

[0017] After performing the channel attention operation on the second eigenvector, multiply it with the second eigenvector to obtain the fourth eigenvector;

[0018] Add the first eigenvector and the third eigenvector to obtain a fifth eigenvector;

[0019] Add the second eigenvector and the fourth eigenvector to obtain a sixth eigenvector;

[0020] Perform an average pooling operation on the fifth eigenvector to obtain a seventh eigenvector;

[0021] Perform a maximum pooling operation on the sixth eigenvector to obtain an eighth eigenvector;

[0022] After performing a difference operation on the seventh eigenvector and the eighth eigenvector, an L2 regularization operation is performed to obtain a feature map with a channel of 1;

[0023] The feature map with channel 1, the fifth eigenvector and the sixth eigenvector are concatenated to obtain a ninth eigenvector. The ninth eigenvector is processed by the third convolutional layer to obtain a tenth eigenvector. The tenth eigenvector is added to the received data to obtain the output of the edge supplement module.

[0024] Furthermore, the data processing process of the superpixel semantic primitive initialization module is as follows:

[0025] The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector;

[0026] Transform the output of the nth decoder to obtain the twelfth eigenvector;

[0027] Transform the eleventh eigenvector to obtain the thirteenth eigenvector;

[0028] The superpixel semantic primitive feature is obtained according to the twelfth eigenvector and the thirteenth eigenvector.

[0029] 2) In a second aspect, the present invention also provides a superpixel semantic primitive generation system, and the specific technical solution is as follows:

[0030] It includes a model building module, a model training module, a target image processing module, a soft association relationship determination module and a super-pixel semantic primitive generation module;

[0031] The model building module is used to: build a superpixel semantic primitive generation model;

[0032] The model training module is used to: train the superpixel semantic primitive generation model to obtain a trained superpixel semantic primitive generation model;

[0033] The target image processing module is used to: process the target image using the trained super-pixel semantic primitive generation model to obtain the super-pixel semantic primitive features of the target image;

[0034] The soft association relationship determination module is used to: establish a soft association relationship between the pixel points of the target image and the superpixel semantic primitives of the target image according to the superpixel semantic primitive features of the target image;

[0035] The superpixel semantic primitive generation module is used to obtain the superpixel semantic primitive of the target image according to the pixel points and soft association relationship of the target image.

[0036] Based on the above solution, the superpixel semantic primitive generation system of the present invention can also be improved as follows.

[0037] Furthermore, the superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged in sequence, n decoders arranged in sequence and n-2 edge supplementation modules; the first encoder is used to receive a sample image, the n encoders and n decoders correspond one to one, and in two adjacent encoders, the output of the front encoder is used as the input of the rear encoder, and in two adjacent decoders, the output of the front decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder, and the output of the second encoder to the n-1th encoder is respectively processed by an edge supplementation module and then input into the corresponding decoder; the output of the nth encoder and the output of the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

[0038] Furthermore, the data processing process of the edge supplement module is:

[0039] The received data is processed by a first convolution layer and a second convolution layer respectively to obtain a first eigenvector and a second eigenvector;

[0040] After performing the channel attention operation on the first eigenvector, multiply it with the first eigenvector to obtain the third eigenvector;

[0041] After performing the channel attention operation on the second eigenvector, multiply it with the second eigenvector to obtain the fourth eigenvector;

[0042] Add the first eigenvector and the third eigenvector to obtain a fifth eigenvector;

[0043] Add the second eigenvector and the fourth eigenvector to obtain a sixth eigenvector;

[0044] Perform an average pooling operation on the fifth eigenvector to obtain a seventh eigenvector;

[0045] Perform a maximum pooling operation on the sixth eigenvector to obtain an eighth eigenvector;

[0046] After performing a difference operation on the seventh eigenvector and the eighth eigenvector, an L2 regularization operation is performed to obtain a feature map with a channel of 1;

[0047] The feature map with channel 1, the fifth eigenvector and the sixth eigenvector are concatenated to obtain a ninth eigenvector. The ninth eigenvector is processed by the third convolutional layer to obtain a tenth eigenvector. The tenth eigenvector is added to the received data to obtain the output of the edge supplement module.

[0048] Furthermore, the data processing process of the superpixel semantic primitive initialization module is as follows:

[0049] The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector;

[0050] Transform the output of the nth decoder to obtain the twelfth eigenvector;

[0051] Transform the eleventh eigenvector to obtain the thirteenth eigenvector;

[0052] The superpixel semantic primitive feature is obtained according to the twelfth eigenvector and the thirteenth eigenvector.

[0053] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, and the at least one computer program being loaded and executed by the processor so that the electronic device implements any one of the above-mentioned superpixel semantic primitive generation methods.

[0054] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any of the above-mentioned superpixel semantic primitive generation methods is implemented.

[0055] The beneficial effects of the present invention are as follows:

[0056] First, for shallow image features, unlike previous methods, the present invention can retain the structural information of complex images to a greater extent by extracting object boundary information from shallow features, making the segmentation boundaries more accurate and clear; secondly, the present invention constructs a deep convolution feature to guide local feature aggregation to adaptively obtain the initial soft clustering center, in the hope of generating an ideal superpixel allocation matrix with fewer iterations, and at the same time uses the allocation matrix to generate superpixel semantic primitives, avoiding the problem of unacceptable time consumption caused by multiple iterative clustering, thereby achieving an improvement in the comprehensive performance in the superpixel semantic primitive generation task; finally, the present invention improves the similarity measurement method, and performs similarity evaluation through the calculation of the improved point product distance, further strengthening the correlation between similar features, and suppressing the interference of features with low correlation on the soft clustering results, thereby obtaining a more accurate allocation matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention:

[0058] Figure 1 A schematic diagram of a process for generating superpixel semantic primitives according to an embodiment of the present invention;

[0059] Figure 2Schematic diagram of the network structure of the superpixel semantic primitive generation model;

[0060] Figure 3 A schematic diagram of the network structure of the edge supplement module;

[0061] Figure 4 Schematic diagram of the network structure of the superpixel semantic primitive initialization module;

[0062] Figure 5 This is a schematic diagram of the introduction to the international standard data set;

[0063] Figure 6 A schematic diagram of the existing LNSNet model, FCN model, Alnet model and SLIC model;

[0064] Figure 7 Comparison data of the number of superpixels of the trained model;

[0065] Figure 8 Comparative data of the boundary recall rate of the trained model;

[0066] Fig. 9 Comparison chart of the visualization results output by multiple trained models;

[0067] Fig.10 A schematic diagram of the structure of a superpixel semantic primitive generation system according to an embodiment of the present invention;

[0068] Fig.11 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0070] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0071] like Figure 1 As shown, a superpixel semantic primitive generation method according to an embodiment of the present invention comprises the following steps:

[0072] S1, build superpixel semantic primitive generation model;

[0073] S2, training the superpixel semantic primitive generation model to obtain a trained superpixel semantic primitive generation model;

[0074] S3, using the trained super-pixel semantic primitive generation model to process the target image to obtain the super-pixel semantic primitive features of the target image;

[0075] S4. According to the super-pixel semantic primitive features of the target image, a soft association relationship is established between the pixel points of the target image and the super-pixel semantic primitives of the target image.

[0076] S5. Obtain a superpixel semantic primitive of the target image according to the pixel points and soft association relationship of the target image.

[0077] Taking the image of each page in the specified bidding document (such as the bidding document of the power industry) as the target image, the trained superpixel semantic primitive generation model can accurately and efficiently extract the superpixel semantic primitive features.

[0078] The target image may also be a road image specified by a user, etc. It should be noted that in order to analyze different target images, the labeled sample images constructed for training the superpixel semantic primitive generation model are different.

[0079] Optionally, in the above technical scheme, the superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged in sequence, n decoders arranged in sequence and n-2 edge supplement modules; the first encoder is used to receive a sample image, the n encoders and n decoders correspond one to one, and in two adjacent encoders, the output of the front encoder is used as the input of the rear encoder, and in two adjacent decoders, the output of the front decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder, and the output of the second encoder to the n-1th encoder is respectively processed by an edge supplement module and then input into the corresponding decoder; the output of the nth encoder and the output of the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

[0080] Wherein, n=3, 4, 5 or 6, etc., can be set according to the actual situation. Taking n=5 as an example, the superpixel semantic primitive generation model is as follows: Figure 2 As shown, Figure 2In the example, “Encoder1” represents the first encoder, “Encoder2” represents the second encoder, “Encoder3” represents the third encoder, “Encoder4” represents the fourth encoder, “Encoder5” represents the fifth encoder, “Decoder1” represents the first decoder, “Decoder2” represents the second decoder, “Decoder3” represents the third decoder, “Decoder34” represents the fourth decoder, and “Decoder5” represents the fifth decoder. The number of edge supplementation modules is 3. The decoder corresponding to the second encoder is the fourth decoder. The output of the second encoder is After processing by the first edge supplement module, we get Will Input the 4th decoder, the decoder corresponding to the 3rd encoder is the 3rd decoder, and the output of the 3rd encoder is After processing by the second edge supplement module, we get Will Input the third decoder, the decoder corresponding to the fourth encoder is the second decoder, and the output of the fourth encoder is After processing by the second edge supplement module, we get Will Enter the second decoder, Figure 2 In the figure, “ES1” represents the first edge supplement module, “ES2” represents the second edge supplement module, “ES3” represents the third edge supplement module, and “Conv+SPI” represents the superpixel semantic primitive initialization module.

[0081] Among them, the output of the first encoder As the input of the 5th decoder, the output of the 5th encoder As the input of the first decoder. Figure 2 The superpixel semantic primitive generation model shown uses an encoder-decoder with skip connections to extract pixel features and initialize superpixel semantic primitive features. The encoder takes a color image as input and generates a deep feature map through a convolutional layer. Then, the decoder gradually upsamples the deep feature map through deconvolution layers to obtain the final pixel features, while also considering the features of the corresponding encoder layer.

[0082] There are 5 encoders in total. Except for the first encoder which only performs feature dimensionality upgrade on the original features of the sample image, the remaining layers halve the width and height of the input feature (previous deep feature map) while increasing the dimension of the feature in the channel. This allows the network to capture broader and more abstract features at each layer while controlling the amount of computation.

[0083] There are 5 decoders in total. The first decoder layer only processes the output of the 5th encoder. The outputs of the second encoder to the n-1 encoder are processed by an edge supplement module and then input to the corresponding decoder. The edge supplement (ES) module is used to extract and supplement the edge information so that the superpixel semantic primitive generation model can have better output results.

[0084] Optionally, in the above technical solution, each edge supplement module includes a first convolution layer, a second convolution layer and a third convolution layer, such as Figure 3 As shown in Figure 2, the data processing process of the edge supplement module is as follows:

[0085] S10, processing the received data through the first convolution layer and the second convolution layer respectively to obtain a first eigenvector and a second eigenvector;

[0086] The data received by the first edge supplement module is the output of the second encoder The data received by the second edge supplement module is the output of the third encoder The data received by the 3rd edge supplement module is the output of the 4th encoder For ease of description, the data received by the edge supplement module is marked as Among them, the value range of x is: 2, 3 and 4, and the first convolutional layer and the second convolutional layer are respectively After processing, two eigenvectors are obtained, specifically the first eigenvector and the second eigenvector

[0087]

[0088] in, Representation: Through the first convolutional layer To process, Representation: Through the second convolutional layer The first convolutional layer and the second convolutional layer respectively include a convolution operation with a convolution kernel size of 3×3, a batch normalization layer, and a ReLU activation layer.

[0089] S11, after performing a channel attention operation on the first eigenvector, multiply it by the first eigenvector to obtain a third eigenvector;

[0090] The first feature vector is Perform channel attention operation to obtain the first eigenvector The attention weight on the channel is then combined with the first eigenvector Multiply them together to get the third eigenvector weighted by channel attention.

[0091]

[0092] in, Representation: For the first eigenvector Perform channel attention operation, Representation: For the first eigenvector The attention weights on the channels and the first eigenvector Multiply the corresponding elements in , that is, multiply the first eigenvector The attention weights on the channels and the first eigenvector Multiply corresponding elements.

[0093] S12, after performing a channel attention operation on the second eigenvector, multiply it by the second eigenvector to obtain a fourth eigenvector;

[0094] The second feature vector is Perform channel attention operation to obtain the second eigenvector The attention weight on the channel is then combined with the second eigenvector Multiply them together to get the fourth eigenvector weighted by channel attention.

[0095]

[0096] in, Representation: For the second eigenvector Perform channel attention operation, Representation: For the second eigenvector The attention weights on the channels and the second eigenvector Multiply the corresponding elements in , that is, multiply the second eigenvector The attention weights on the channels and the second eigenvector Multiply corresponding elements.

[0097] S13, adding the first eigenvector and the third eigenvector to obtain a fifth eigenvector, specifically:

[0098] For the first eigenvector and the third eigenvector Add together to get the fifth eigenvector with channel attention weights

[0099]

[0100] in, Representation: For the first eigenvector and the third eigenvector Add the corresponding elements in , that is, add the first eigenvector and the third eigenvector Add corresponding elements.

[0101] S14, adding the second eigenvector and the fourth eigenvector to obtain a sixth eigenvector, specifically:

[0102] For the second eigenvector and the fourth eigenvector Add together to get the sixth eigenvector with channel attention weights

[0103]

[0104] in, Representation: For the second eigenvector and the fourth eigenvector Add the corresponding elements in , that is, the second eigenvector and the fourth eigenvector Add corresponding elements.

[0105] S15, performing an average pooling operation on the fifth eigenvector to obtain a seventh eigenvector, specifically:

[0106] For the fifth eigenvector Perform an average pooling operation to obtain the seventh eigenvector.

[0107] S16, performing a maximum pooling operation on the sixth eigenvector to obtain an eighth eigenvector, specifically:

[0108] For the sixth eigenvector Perform the maximum pooling operation to obtain the eighth eigenvector.

[0109] After performing the difference operation on the seventh eigenvector and the eighth eigenvector, an L2 regularization operation is performed to obtain a feature map with a channel of 1. The feature map with a channel of 1 is a significant edge information, which is recorded as

[0110]

[0111] in, Representation: For the fifth eigenvector Perform an average pooling operation, that is, the seventh eigenvector is The calculated eigenvector is Representation: For the sixth eigenvector Perform the maximum pooling operation, that is, the eighth eigenvector is The calculated eigenvector is Means: perform the difference operation on the seventh eigenvector and the eighth eigenvector, Indicates: Yes Perform L2 regularization operation.

[0112] S17, concatenate the feature map with channel 1, the fifth eigenvector and the sixth eigenvector to obtain a ninth eigenvector, process the ninth eigenvector through the third convolutional layer to obtain a tenth eigenvector, add the tenth eigenvector to the received data, and obtain the output of the edge supplementation module This is achieved specifically through the following formula:

[0113]

[0114] in, Represents: the output of the edge supplementation module, Indicates: and Splicing is performed on the channel dimension, and the ninth eigenvector is The calculated eigenvector is Representation: Through the third convolutional layer After processing, the tenth eigenvector is The calculated eigenvector is Representation: Substitute the tenth eigenvector with the first eigenvector In addition, the third convolutional layer includes a convolution operation with a kernel size of 3×3, a batch normalization layer, and a ReLU activation layer.

[0115] Traverse the output of the second encoder The output of the third encoder and the output of the 4th encoder That is, traverse x=2, x=3 and x=4 to get the output of the first edge supplement module Output of the second edge supplement module and the output of the third edge supplement module

[0116] Optionally, in the above technical solution, the superpixel semantic primitive initialization module includes a fourth convolutional layer, such as Figure 4 As shown in Figure 2, the data processing process of the superpixel semantic primitive initialization module is as follows:

[0117] S20, the output of the nth encoder is processed by the fourth convolutional layer to obtain an eleventh feature vector, specifically:

[0118] When n=5, the output of the 5th encoder After being processed by the fourth convolutional layer, the output of the fifth encoder is Dimension reduction, get the eleventh eigenvector F g , the eleventh eigenvector F g The eleventh eigenvector F has the same number of channels as the output of the fifth decoder. g Can guide the generation of superpixel semantic primitive features, the eleventh feature vector F g The acquisition process is implemented through the following expression:

[0119]

[0120] in, Represents: The output of the 5th encoder through the 4th convolutional layer Perform dimensionality reduction processing.

[0121] S21, transform the output of the nth decoder to obtain the twelfth eigenvector, specifically:

[0122] The output of the nth decoder is the pixel feature, denoted by F p , F p ∈R H×W×C , where H represents the height of the sample image, W represents the width of the training sample image, and C represents the number of channels of the pixel features output by the nth decoder, which is also the standard number of channels in the process of calculating the soft association relationship. p Transform and get the twelfth eigenvector F p ′ , Among them, h num × num Indicates: the number of superpixel semantic primitives that need to be segmented, h num Indicates: In the initial state, the height of the feature map composed of superpixel semantic primitives, w num Indicates: In the initial state, the width of the feature map composed of superpixel semantic primitives, h sp × sp Indicates: In the initial state, the number of pixels in the pixel block contained in each superpixel semantic primitive, h sp Indicates: In the initial state, the height of the pixel block contained in each superpixel semantic primitive, w sp Represents: In the initial state, the width of the pixel block contained in each superpixel semantic primitive, and satisfies the following conditions:

[0123] H=h num ×h sp

[0124] W=w num × sp

[0125] When actually training the model, H and h sp are fixed values, W and w sp All are fixed values.

[0126] S22. Transform the eleventh eigenvector to obtain a thirteenth eigenvector.

[0127] Among them, the eleventh eigenvector F g Transform and get the thirteenth eigenvector F g ′ ,

[0128] S23, according to the twelfth eigenvector and the thirteenth eigenvector, obtain the superpixel semantic primitive feature, which is specifically implemented by the following formula:

[0129]

[0130] Among them, F sp Representation: Superpixel semantic primitive features, Linear1(F g ′ ) means: the thirteenth eigenvector F is g ′ To process, Representation: The twelfth eigenvector F is transformed by the second linear layer p ′ For processing, T represents matrix transpose, Indicates: Yes Perform SoftMax operation.

[0131] Optionally, the specific implementation process of S4 is as follows:

[0132] According to the superpixel semantic primitive features of the target image, a soft association relationship between the pixel points of the target image and the superpixel semantic primitives of the target image is established by using a differentiable soft clustering method, specifically:

[0133] By calculating the similarity between each pixel of the target image and the nine surrounding superpixel semantic primitives, a soft association relationship between the pixel of the target image and the superpixel semantic primitive of the target image is established. The soft association relationship is specifically represented by the allocation matrix Q:

[0134]

[0135]

[0136] Among them, F p ″ means: after the target image is input into the trained superpixel semantic primitive generation model, the feature vector of the output of the nth decoder of the trained superpixel semantic primitive generation model is converted. It can be simply considered that F p ″ is the twelfth eigenvector corresponding to the target image, F s ′ p ′ Representation: Superpixel semantic primitive features F of the target image s ′ p The transformed feature vector is Represents: Each superpixel semantic primitive contains the features of the nine surrounding superpixel semantic primitives including itself, (·) 2 represents the operation of finding the second power, ⊙ represents the multiplication of corresponding positions, Sgn(x) is a sign function used to retain the positive and negative signs of the input parameter x, x is the input parameter, and the sign function will return the positive and negative signs of the input parameter x, Sgn(F p ″·(F s ′ p ′ ) T ) indicates that F is reserved p ″·(F s ′ p ′ ) T The calculated result is positive or negative (because the second half of the allocation matrix Q involves a square operation, resulting in the loss of F p ″·(F s ′ p ′ ) T The sign of the calculated result).

[0137] In this method, the specific steps of using a traditional convolutional neural network to extract features from a given image and obtain image features at different extraction stages include:

[0138] The input is an RGB color image, and a five-stage small "U"-shaped network is constructed with channel dimensions of [16, 32, 64, 128, 256], extracting the deepest features as the seed guidance features of the superpixel clustering center, and the final decoder output is used as the feature of the pixel to be clustered.

[0139] Optionally, in the above technical solution, the specific steps of using a uniquely designed edge extractor to extract edges from a given shallow convolution feature and obtain edge enhancement for a convolutional network include:

[0140] The input feature map is passed through two different 3x3 convolution layers to obtain two feature maps; then the channel attention layer is used to select the dimension containing edge information, and then the average pooling layer and the maximum pooling layer are used to subtract the two results. L2 regularization is used on the channel dimension to obtain the boundary map, and then the boundary map is upsampled, concatenated with the two feature maps and convolved back to the original input feature map dimension through 1x1 convolution, and the residual connection is used for the feature map originally input.

[0141] Optionally, in the above technical solution, through a uniquely designed attention mechanism, the deepest features of the neural network are used to guide the output features of the neural network to aggregate, so as to obtain the initial superpixel semantic primitive seeds, and the specific steps include:

[0142] First, this part includes two dimensionality reduction layers, which use a 1×1 convolution kernel for the superpixel semantic primitive seed feature and supplement it with batch normalization (BatchNormalization, BN) and ReLU activation function to make its dimension the same as the pixel feature dimension.

[0143] The pixel features are divided into small squares with a step size of 16. The number of squares is the same as the number of superpixel semantic primitive seed features. Each attention module receives a superpixel semantic primitive seed feature and a 16x16 pixel feature at the corresponding position as input, and uses the seed feature to select and aggregate pixel features.

[0144] Through the improved similarity evaluation method, the superpixel semantic primitive seed and pixel information similarity measurement matrix are generated to obtain the final soft assignment matrix. The specific steps include:

[0145] The pixel features are multiplied and accumulated with the corresponding channels of the nine adjacent superpixel semantic primitive seeds, and the result is squared and multiplied by the sign function (Sgn). This is used as the basis for similarity measurement to generate a similarity matrix with 9 channels. Normalization operation (Softmax) is performed on the channel dimension, and the pixels are assigned to the surrounding nine superpixel semantic primitives according to percentage. The same operation is performed on all pixels to obtain a superpixel soft assignment matrix based on the improved similarity measurement distance.

[0146] Optionally, in the above technical solution, the soft assignment matrix is ​​used to generate and reconstruct the object label information, pixel coordinate information, RGB color information of the semantic segmentation data set and the original corresponding information to construct cross entropy loss optimization.

[0147] Optionally, in the above technical solution, the selected optimization method is an adaptive moment estimation algorithm (Adam) in a deep learning framework (Pytorch).

[0148] In the present invention, a traditional convolutional neural network is used to extract features from a given image and obtain image features at different extraction stages. A uniquely designed edge extractor is used to extract edges from a given shallow convolution feature to enhance the edge information of the convolution feature. Through a uniquely designed attention mechanism, the deepest features of the neural network are used to guide the output features of the neural network to aggregate and obtain the initial superpixel semantic primitive seeds. Through an improved similarity evaluation method, superpixel semantic primitive seeds and pixel information similarity measurement matrices are generated to obtain the final soft assignment matrix, and the soft assignment matrix is ​​used to generate semantic primitives.

[0149] The present invention constructs a soft clustering superpixel semantic primitive generation model based on edge enhancement strategy to perform superpixel segmentation on a given arbitrary color image and generate the features of superpixel semantic primitives. To this end, the present invention first uses a 5-stage "U"-shaped network structure image feature extraction channel, in which the input end (left side) is an encoder, the output end (right side) is a decoder, and the corresponding layers of the networks on both sides are skipped and connected, and the image is feature extracted through this small convolutional neural network. Specifically, firstly, for shallow image features, unlike previous methods, we can retain the structural information of complex images to a greater extent by extracting the object boundary information in the shallow features, so that the segmentation boundary is more accurate and clear; secondly, the present invention constructs a deep convolution feature to guide the aggregation of local features to adaptively obtain the initial soft clustering center, in the hope of generating an ideal superpixel allocation matrix with fewer iterations, and at the same time uses the allocation matrix to generate superpixel semantic primitives, avoiding the problem of unacceptable time consumption caused by multiple iterations of clustering, thereby achieving the improvement of comprehensive performance in the superpixel semantic primitive generation task; finally, the present invention improves the similarity measurement method, and evaluates the similarity by calculating the improved point multiplication distance, further strengthening the correlation between similar features, and suppressing the interference of features with low correlation on the soft clustering results, thereby obtaining a more accurate allocation matrix. The shallow feature boundary extractor constructed by the present invention can extract clear object boundary information. Comprehensive experiments have proved that the edge information enhancement module of the present invention can effectively improve the representation ability of pixel features.

[0150] The present invention proposes a novel attention-enhanced superpixel semantic primitive feature initialization construction module, which can gradually and adaptively complete the calculation of superpixel semantic primitive seed features. We innovatively use the deepest convolutional features rich in global information, and input them into the attention module as the guidance information of the superpixel semantic primitive features, to guide the aggregation of pixel features and construct superpixel semantic primitive seed features. Through experiments, it can be verified that the model provided by the present invention has a stronger superpixel semantic primitive seed extraction capability, and can obtain good superpixel semantic primitive generation results with a small number of iterations.

[0151] The final pixel allocation matrix of the model proposed in the present invention is obtained by the improved vector dot multiplication soft clustering method, and the module function will not be completely invalidated due to the change of the oriented task, that is, after being connected to the multimodal document processing framework, the model can continue to be trained with it and ensure sufficiently good superpixel semantic primitive generation results. Therefore, after we pre-train the model, the model can be directly embedded in the multimodal document processing framework without explicit constraints.

[0152] The superpixel semantic primitive generation model in the present invention and the existing LNSNet model, FCN model, Alnet model and SLIC model are trained using international standard data sets, and the segmentation performance of the trained models is evaluated using multiple evaluation indicators, wherein the international standard data set can be the BSDS500 data set. The source of the international standard data set is as follows Figure 5 shown.

[0153] Among them, the sources of the existing LNSNet model, FCN model, Alnet model and SLIC model are as follows Figure 6 shown.

[0154] Among them, multiple evaluation indicators include: number of superpixels and boundary recall rate, Figure 7 Comparative data of the number of superpixels of the trained models are shown. Figure 8 The comparison data of the boundary recall rate of the trained model is shown. Fig. 9 The visualization results output by multiple trained models are shown. Figure 7 , Figure 8 and Fig. 9 The "ours" in the above-mentioned embodiment represents the trained superpixel semantic primitive generation model in the present invention. In the above-mentioned embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above-mentioned embodiments may be included.

[0155] like Fig.10As shown, a superpixel semantic primitive generation system 200 according to an embodiment of the present invention includes a model building module 201, a model training module 202, a target image processing module 203, a soft association relationship determination module 204 and a superpixel semantic primitive generation module 205;

[0156] The model building module 201 is used to: build a superpixel semantic primitive generation model;

[0157] The model training module 202 is used to: train the superpixel semantic primitive generation model to obtain a trained superpixel semantic primitive generation model;

[0158] The target image processing module 203 is used to: process the target image using the trained super-pixel semantic primitive generation model to obtain the super-pixel semantic primitive features of the target image;

[0159] The soft association relationship determination module 204 is used to: establish a soft association relationship between the pixel points of the target image and the superpixel semantic primitives of the target image according to the superpixel semantic primitive features of the target image;

[0160] The super-pixel semantic primitive generation module 205 is used to obtain the super-pixel semantic primitive of the target image according to the pixel points and soft association relationship of the target image.

[0161] Optionally, in the above technical solution, the superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged in sequence, n decoders arranged in sequence and n-2 edge supplementation modules; the first encoder is used to receive a sample image, the n encoders and n decoders correspond one to one, and in two adjacent encoders, the output of the front encoder is used as the input of the rear encoder, in two adjacent decoders, the output of the front decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder, and the output of the second encoder to the n-1th encoder is respectively processed by an edge supplementation module and then input to the corresponding decoder;

[0162] The output of the nth encoder and the output of the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

[0163] Optionally, in the above technical solution, the data processing process of the edge supplement module is:

[0164] The received data is processed by the first convolution layer and the second convolution layer respectively to obtain the first eigenvector and the second eigenvector; the first eigenvector is subjected to a channel attention operation and then multiplied with the first eigenvector to obtain the third eigenvector; the second eigenvector is subjected to a channel attention operation and then multiplied with the second eigenvector to obtain the fourth eigenvector; the first eigenvector and the third eigenvector are added to obtain the fifth eigenvector; the second eigenvector and the fourth eigenvector are added to obtain the sixth eigenvector; the fifth eigenvector is subjected to an average pooling operation to obtain the seventh eigenvector; the sixth eigenvector is subjected to a maximum pooling operation to obtain the eighth eigenvector; the seventh eigenvector and the eighth eigenvector are subjected to a difference operation and then an L2 regularization operation to obtain a feature map with a channel of 1; the feature map with a channel of 1, the fifth eigenvector and the sixth eigenvector are concatenated to obtain a ninth eigenvector; the ninth eigenvector is subjected to processing by the third convolution layer to obtain a tenth eigenvector; the tenth eigenvector is added to the received data to obtain the output of the edge supplementation module.

[0165] Optionally, in the above technical solution, the data processing process of the superpixel semantic primitive initialization module is:

[0166] The output of the nth encoder is processed by the fourth convolutional layer to obtain an eleventh eigenvector; the output of the nth decoder is transformed to obtain a twelfth eigenvector, and the eleventh eigenvector is transformed to obtain a thirteenth eigenvector; the superpixel semantic primitive feature is obtained according to the twelfth eigenvector and the thirteenth eigenvector.

[0167] It should be noted that the beneficial effects of the superpixel semantic primitive generation system 200 provided in the above embodiment are the same as the beneficial effects of the superpixel semantic primitive generation method described above, which will not be repeated here. In addition, when the system provided in the above embodiment realizes its functions, it only uses the division of the above-mentioned functional modules as an example. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0168] Among them, the superpixel semantic primitive generation system of the present invention can be a computer program (including program code) running in a computer device. For example, the superpixel semantic primitive generation system of the present invention is an application software that can be used to execute the corresponding steps in the superpixel semantic primitive generation method of the present invention.

[0169] In some embodiments, the superpixel semantic primitive generation system of the present invention can be implemented by a combination of software and hardware. As an example, the superpixel semantic primitive generation system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the superpixel semantic primitive generation method of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0170] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0171] An electronic device according to an embodiment of the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned superpixel semantic primitive generation methods is implemented. That is to say, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the superpixel semantic primitive generation method shown in any embodiment of the present invention by calling the computer program.

[0172] In an alternative embodiment, an electronic device is provided, such as Fig.11 As shown, Fig.11 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0173] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0174] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 In the figure, only one thick line is used to represent the bus 4002, but this does not mean that there is only one bus or one type of bus.

[0175] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0176] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0177] Among them, the electronic device can also be a terminal device, and the terminal device can be any device that can install applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0178] It should be noted that Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0179] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned superpixel semantic primitive generation methods is implemented.

[0180] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0181] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned superpixel semantic primitive generation.

[0182] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0183] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0185] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0186] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0187] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0188] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0189] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A superpixel semantic primitive generation method, characterized in that: include: Construct a superpixel semantic primitive generation model; Training the superpixel semantic primitive generation model to obtain a trained superpixel semantic primitive generation model; The target image is processed using the trained super-pixel semantic primitive generation model to obtain the super-pixel semantic primitive features of the target image; According to the superpixel semantic primitive features of the target image, a soft association relationship is established between the pixel points of the target image and the superpixel semantic primitives of the target image; A superpixel semantic primitive of the target image is obtained according to the pixel points of the target image and the soft association relationship.

2. The method for generating superpixel semantic primitives according to claim 1, characterized in that: The superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged in sequence, n decoders arranged in sequence and n-2 edge supplement modules; the first encoder is used to receive a sample image, the n encoders and n decoders correspond one to one, in two adjacent encoders, the output of the front encoder is used as the input of the rear encoder, in two adjacent decoders, the output of the front decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder, and the output of the second encoder to the n-1th encoder is respectively processed by an edge supplement module and then input to the corresponding decoder; the output of the nth encoder and the output of the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

3. A superpixel semantic primitive generation method according to claim 2, characterized in that: The data processing process of the edge supplement module is as follows: The received data is processed by a first convolution layer and a second convolution layer respectively to obtain a first eigenvector and a second eigenvector; After performing a channel attention operation on the first feature vector, multiplying the first feature vector with the first feature vector to obtain a third feature vector; After performing a channel attention operation on the second eigenvector, multiplying the second eigenvector by the second eigenvector to obtain a fourth eigenvector; Adding the first eigenvector and the third eigenvector to obtain a fifth eigenvector; Adding the second eigenvector and the fourth eigenvector to obtain a sixth eigenvector; Performing an average pooling operation on the fifth eigenvector to obtain a seventh eigenvector; Performing a maximum pooling operation on the sixth eigenvector to obtain an eighth eigenvector; After performing a difference operation on the seventh eigenvector and the eighth eigenvector, an L2 regularization operation is performed to obtain a feature map with a channel of 1; The feature map with channel 1, the fifth eigenvector and the sixth eigenvector are concatenated to obtain a ninth eigenvector, the ninth eigenvector is processed by the third convolutional layer to obtain a tenth eigenvector, and the tenth eigenvector is added to the received data to obtain the output of the edge supplement module.

4. The method for generating superpixel semantic primitives according to claim 3, characterized in that: The data processing process of the superpixel semantic primitive initialization module is as follows: The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector; Transform the output of the nth decoder to obtain the twelfth eigenvector; Transforming the eleventh eigenvector to obtain a thirteenth eigenvector; The superpixel semantic primitive feature is obtained according to the twelfth eigenvector and the thirteenth eigenvector.

5. A superpixel semantic primitive generation system, characterized in that: It includes a model building module, a model training module, a target image processing module, a soft association relationship determination module and a super-pixel semantic primitive generation module; The model building module is used to: build a superpixel semantic primitive generation model; The model training module is used to: train the superpixel semantic primitive generation model to obtain a trained superpixel semantic primitive generation model; The target image processing module is used to: process the target image using the trained super-pixel semantic primitive generation model to obtain the super-pixel semantic primitive features of the target image; The soft association relationship determination module is used to: establish a soft association relationship between the pixel points of the target image and the superpixel semantic primitives of the target image according to the superpixel semantic primitive features of the target image; The superpixel semantic primitive generation module is used to obtain the superpixel semantic primitive of the target image according to the pixel points of the target image and the soft association relationship.

6. A superpixel semantic primitive generation system according to claim 5, characterized in that: The superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged in sequence, n decoders arranged in sequence and n-2 edge supplement modules; the first encoder is used to receive a sample image, the n encoders and n decoders correspond one to one, in two adjacent encoders, the output of the front encoder is used as the input of the rear encoder, in two adjacent decoders, the output of the front decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder, and the output of the second encoder to the n-1th encoder is respectively processed by an edge supplement module and then input to the corresponding decoder; the output of the nth encoder and the output of the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

7. A superpixel semantic primitive generation system according to claim 6, characterized in that: The data processing process of the edge supplement module is as follows: The received data is processed by a first convolution layer and a second convolution layer respectively to obtain a first eigenvector and a second eigenvector; After performing a channel attention operation on the first feature vector, multiplying the first feature vector with the first feature vector to obtain a third feature vector; After performing a channel attention operation on the second eigenvector, multiplying the second eigenvector by the second eigenvector to obtain a fourth eigenvector; Adding the first eigenvector and the third eigenvector to obtain a fifth eigenvector; Adding the second eigenvector and the fourth eigenvector to obtain a sixth eigenvector; Performing an average pooling operation on the fifth eigenvector to obtain a seventh eigenvector; Performing a maximum pooling operation on the sixth eigenvector to obtain an eighth eigenvector; After performing a difference operation on the seventh eigenvector and the eighth eigenvector, an L2 regularization operation is performed to obtain a feature map with a channel of 1; The feature map with channel 1, the fifth eigenvector and the sixth eigenvector are concatenated to obtain a ninth eigenvector, the ninth eigenvector is processed by the third convolutional layer to obtain a tenth eigenvector, and the tenth eigenvector is added to the received data to obtain the output of the edge supplement module.

8. A superpixel semantic primitive generation system according to claim 7, characterized in that: The data processing process of the superpixel semantic primitive initialization module is as follows: The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector; Transform the output of the nth decoder to obtain the twelfth eigenvector; Transforming the eleventh eigenvector to obtain a thirteenth eigenvector; The superpixel semantic primitive feature is obtained according to the twelfth eigenvector and the thirteenth eigenvector.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a superpixel semantic primitive generation method as claimed in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating superpixel semantic primitives according to any one of claims 1 to 4 is implemented.

Images