Superpixel semantic primitive generation method and system, electronic device and storage medium

Abstract

The application discloses a kind of superpixel semantic primitive generation method, system, electronic equipment and storage medium, it is related to data processing technical field, method includes: constructing superpixel semantic primitive generation model;Superpixel semantic primitive generation model is trained, and the superpixel semantic primitive generation model of training is obtained;Utilize the superpixel semantic primitive generation model of training to process target image, and the superpixel semantic primitive feature of target image is obtained;According to the superpixel semantic primitive feature of target image, the soft association between the pixel point of target image and the superpixel semantic primitive of target image is established;According to the pixel point of target image and soft association, the superpixel semantic primitive of target image is obtained.The superpixel semantic primitive of the present application can be efficiently and accurately generated.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, electronic device, and storage medium for generating superpixel semantic primitives. Background Technology

[0002] With the rapid development of the internet and digital technologies, and the rapid increase in multimodal information, traditional information retrieval and processing methods based on a single modality (such as plain text) often fall short when faced with multimodal documents. To effectively address this challenge, research on multimodal document information processing has emerged. Multimodal documents typically include multiple information sources such as text, images, and charts, which jointly convey the core content of the document through contextual semantic connections. Images not only supplement textual information but also often directly express the core of the document. Therefore, how to efficiently and accurately extract semantic primitives from images that can be used for document understanding has become crucial for improving document information retrieval and automatic understanding capabilities. Superpixel segmentation, as an effective method to reduce the number of pixel-level primitives in an image and divide it into pixel blocks rich in semantic information, can provide more perspectives for generating semantic primitives for multimodal document images. Superpixel segmentation is a method that aggregates similar pixels into larger units—superpixels. The core advantage of this method is that it can reduce the complexity of image data while preserving important edge and structural information of the image to a certain extent.

[0003] Currently, many superpixel algorithms based on traditional image processing and machine learning techniques have been proposed and widely applied. However, most of these traditional algorithms are difficult to parallelize, making them hard to integrate into existing large-scale multimodal document processing frameworks. In recent years, researchers have begun to utilize deep network learning techniques to improve superpixel segmentation performance. The advantage of this approach is that it can embed the superpixel segmentation process into a large-scale parallel processing framework and generate superpixel semantic primitives of the image during segmentation, thereby providing more accurate semantic support for subsequent understanding and analysis. However, existing models struggle to balance computational efficiency, boundary adhesion, and robustness. Therefore, designing a deep learning-based superpixel semantic primitive generation network that reduces the computational cost of superpixel segmentation while improving segmentation accuracy and the quality of semantic primitive features is one of the current research challenges.

[0004] In superpixel-based multimodal image semantic primitive generation, it is necessary not only to consider how to extract features from each pixel, but also how to assign pixels to corresponding superpixels using these extracted features. For feature extraction, many current methods either directly use weighted sums of initial image features or employ convolutional neural networks, neglecting the role of edges. In fact, edges are among the most important features in an image, depicting the contours and shapes of objects. Paying attention to image boundaries during semantic primitive generation ensures the integrity of objects in the image, preventing pixels belonging to different objects from being incorrectly assigned to the same superpixel semantic primitive. Current methods for assigning pixels to corresponding superpixel semantic primitives include graph partitioning, clustering, region evolution, and deep learning. The first three methods suffer from low computational efficiency and difficulty in parallelization; deep learning-based methods are fast but require high-quality labeled data and have poor generalization performance. Therefore, how to fully utilize the edge features of image objects and how to accurately and stably generate superpixel semantic primitives are of great research significance, and this also presents certain challenges to the further development of multimodal document processing. Summary of the Invention

[0005] The technical problem to be solved by this invention is to address the shortcomings of existing technologies. Specifically, it provides a method, system, electronic device, and storage medium for generating superpixel semantic primitives, as detailed below:

[0006] 1) In a first aspect, the present invention provides a method for generating superpixel semantic primitives, the specific technical solution of which is as follows:

[0007] Construct a superpixel semantic primitive generation model;

[0008] The superpixel semantic primitive generation model is trained to obtain the trained superpixel semantic primitive generation model.

[0009] The target image is processed using a trained superpixel semantic primitive generation model to obtain the superpixel semantic primitive features of the target image;

[0010] Based on the superpixel semantic primitive features of the target image, a soft association relationship is established between the pixels of the target image and the superpixel semantic primitives of the target image.

[0011] Based on the pixels of the target image and the soft correlation relationships, the superpixel semantic primitives of the target image are obtained.

[0012] Based on the above scheme, the superpixel semantic primitive generation method of the present invention can be further improved as follows.

[0013] Furthermore, the superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders set sequentially, n decoders set sequentially, and n-2 edge completion modules; the first encoder is used to receive the sample image, the n encoders and n decoders correspond one-to-one, in two adjacent encoders, the output of the previous encoder is used as the input of the next encoder, in two adjacent decoders, the output of the previous decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder. The outputs of the second to the (n-1)th encoders are processed by an edge completion module before being input to the corresponding decoder; the outputs of the nth encoder and the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

[0014] Furthermore, the data processing procedure of the edge supplementation module is as follows:

[0015] The received data is processed by the first convolutional layer and the second convolutional layer respectively to obtain the first feature vector and the second feature vector;

[0016] After performing channel attention operation on the first feature vector, multiply it by the first feature vector to obtain the third feature vector;

[0017] After performing channel attention operation on the second feature vector, multiply it by the second feature vector to obtain the fourth feature vector;

[0018] Add the first eigenvector and the third eigenvector to obtain the fifth eigenvector;

[0019] Add the second and fourth eigenvectors to obtain the sixth eigenvector;

[0020] The fifth feature vector is subjected to average pooling to obtain the seventh feature vector.

[0021] Perform max pooling on the sixth feature vector to obtain the eighth feature vector;

[0022] After subtracting the seventh and eighth eigenvectors, L2 regularization is performed to obtain a feature map with 1 channel.

[0023] The feature map with channel 1, the fifth feature vector, and the sixth feature vector are concatenated to obtain the ninth feature vector. The ninth feature vector is then processed by the third convolutional layer to obtain the tenth feature vector. The tenth feature vector is then added to the received data to obtain the output of the edge supplementation module.

[0024] Furthermore, the data processing procedure of the superpixel semantic primitive initialization module is as follows:

[0025] The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector;

[0026] Transform the output of the nth decoder to obtain the twelfth feature vector;

[0027] Transform the eleventh eigenvector to obtain the thirteenth eigenvector;

[0028] The superpixel semantic primitive features are obtained based on the twelfth and thirteenth feature vectors.

[0029] 2) In a second aspect, the present invention also provides a superpixel semantic primitive generation system, the specific technical solution of which is as follows:

[0030] It includes a model building module, a model training module, a target image processing module, a soft relation determination module, and a superpixel semantic primitive generation module;

[0031] The model building module is used to: build superpixel semantic primitive generation models;

[0032] The model training module is used to train the superpixel semantic primitive generation model to obtain the trained superpixel semantic primitive generation model.

[0033] The target image processing module is used to: process the target image using the trained superpixel semantic primitive generation model to obtain the superpixel semantic primitive features of the target image;

[0034] The soft association determination module is used to: establish soft association relationships between pixels of the target image and superpixel semantic primitives based on the superpixel semantic primitive features of the target image;

[0035] The superpixel semantic primitive generation module is used to: obtain the superpixel semantic primitives of the target image based on the pixels of the target image and the soft correlation relationships.

[0036] Based on the above scheme, the superpixel semantic primitive generation system of the present invention can be further improved as follows.

[0037] Furthermore, the superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders set sequentially, n decoders set sequentially, and n-2 edge completion modules; the first encoder is used to receive the sample image, the n encoders and n decoders correspond one-to-one, in two adjacent encoders, the output of the previous encoder is used as the input of the next encoder, in two adjacent decoders, the output of the previous decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder. The outputs of the second to the (n-1)th encoders are processed by an edge completion module before being input to the corresponding decoder; the outputs of the nth encoder and the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

[0038] Furthermore, the data processing procedure of the edge supplementation module is as follows:

[0039] The received data is processed by the first convolutional layer and the second convolutional layer respectively to obtain the first feature vector and the second feature vector;

[0040] After performing channel attention operation on the first feature vector, multiply it by the first feature vector to obtain the third feature vector;

[0041] After performing channel attention operation on the second feature vector, multiply it by the second feature vector to obtain the fourth feature vector;

[0042] Add the first eigenvector and the third eigenvector to obtain the fifth eigenvector;

[0043] Add the second and fourth eigenvectors to obtain the sixth eigenvector;

[0044] The fifth feature vector is subjected to average pooling to obtain the seventh feature vector.

[0045] Perform max pooling on the sixth feature vector to obtain the eighth feature vector;

[0046] After subtracting the seventh and eighth eigenvectors, L2 regularization is performed to obtain a feature map with 1 channel.

[0047] The feature map with channel 1, the fifth feature vector, and the sixth feature vector are concatenated to obtain the ninth feature vector. The ninth feature vector is then processed by the third convolutional layer to obtain the tenth feature vector. The tenth feature vector is then added to the received data to obtain the output of the edge supplementation module.

[0048] Furthermore, the data processing procedure of the superpixel semantic primitive initialization module is as follows:

[0049] The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector;

[0050] Transform the output of the nth decoder to obtain the twelfth feature vector;

[0051] Transform the eleventh eigenvector to obtain the thirteenth eigenvector;

[0052] The superpixel semantic primitive features are obtained based on the twelfth and thirteenth feature vectors.

[0053] 3) In a third aspect, the present invention also provides an electronic device, the electronic device including a processor coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the electronic device to implement any of the above-described superpixel semantic primitive generation methods.

[0054] 4) In a fourth aspect, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described superpixel semantic primitive generation methods.

[0055] The beneficial effects of this invention are as follows:

[0056] First, unlike previous methods, this invention extracts object boundary information from shallow image features, thus preserving more of the structural information of complex images and making the segmentation boundaries more accurate and clear. Second, this invention constructs a deep convolutional feature to guide local feature aggregation, thereby adaptively obtaining initial soft cluster centers. This aims to generate an ideal superpixel allocation matrix with fewer iterations. Simultaneously, the allocation matrix is ​​used to generate superpixel semantic primitives, avoiding the unacceptable time consumption caused by multiple iterations of clustering, thereby improving the overall performance in the superpixel semantic primitive generation task. Finally, this invention improves the similarity measurement method by using a modified dot product distance calculation for similarity evaluation, further strengthening the correlation between similar features and suppressing the interference of features with low correlation on the soft clustering results, thus obtaining a more accurate allocation matrix. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below:

[0058] Figure 1 This is a flowchart illustrating a method for generating superpixel semantic primitives according to an embodiment of the present invention.

[0059] Figure 2A schematic diagram of the network structure of the superpixel semantic primitive generation model;

[0060] Figure 3 A schematic diagram of the network structure for edge supplementary modules;

[0061] Figure 4 A schematic diagram of the network structure for the superpixel semantic primitive initialization module;

[0062] Figure 5 A schematic diagram illustrating the international standard dataset;

[0063] Figure 6 A brief illustration of the existing LNSNet, FCN, Alnet, and SLIC models;

[0064] Figure 7 Comparative data on the number of superpixels of the trained models;

[0065] Figure 8 Comparative data for the boundary recall of the trained model;

[0066] Figure 9 A comparison chart of the output results of multiple trained models;

[0067] Figure 10 This is a schematic diagram of the structure of a superpixel semantic primitive generation system according to an embodiment of the present invention;

[0068] Figure 11 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0069] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0070] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0071] like Figure 1 As shown, an embodiment of the present invention provides a method for generating superpixel semantic primitives, which includes the following steps:

[0072] S1. Construct a superpixel semantic primitive generation model;

[0073] S2. Train the superpixel semantic primitive generation model to obtain the trained superpixel semantic primitive generation model.

[0074] S3. Use the trained superpixel semantic primitive generation model to process the target image and obtain the superpixel semantic primitive features of the target image.

[0075] S4. Based on the superpixel semantic primitive features of the target image, establish a soft association relationship between the pixels of the target image and the superpixel semantic primitives of the target image.

[0076] S5. Based on the pixels of the target image and the soft correlation relationships, obtain the superpixel semantic primitives of the target image.

[0077] Using the images of each page in a specified bidding document (such as a bidding document in the power industry) as the target images, a trained superpixel semantic primitive generation model can accurately and efficiently extract superpixel semantic primitive features.

[0078] The target image can be a road image specified by the user. It should be noted that the labeled sample images constructed by training the superpixel semantic primitive generation model are different for analyzing different target images.

[0079] Optionally, in the above technical solution, the superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged sequentially, n decoders arranged sequentially, and n-2 edge supplementation modules; the first encoder is used to receive the sample image, the n encoders and n decoders correspond one-to-one, in two adjacent encoders, the output of the previous encoder is used as the input of the next encoder, in two adjacent decoders, the output of the previous decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder; the outputs of the second encoder to the (n-1)th encoder are processed by an edge supplementation module and then input to the corresponding decoder; the outputs of the nth encoder and the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

[0080] Where n = 3, 4, 5, or 6, etc., can be set according to the actual situation. Taking n = 5 as an example, the superpixel semantic primitive generation model is as follows: Figure 2 As shown, Figure 2In this code, "Encoder1" represents the first encoder, "Encoder2" the second encoder, "Encoder3" the third encoder, "Encoder4" the fourth encoder, and "Encoder5" the fifth encoder; "Decoder1" the first decoder, "Decoder2" the second decoder, "Decoder3" the third decoder, "Decoder4" the fourth decoder, and "Decoder5" the fifth decoder. There are three edge-padding modules. The decoder corresponding to the second encoder is the fourth decoder. The output of the second encoder... After processing by the first edge completion module, the result is... Will The input is the fourth decoder, the decoder corresponding to the third encoder is the third decoder, and the output of the third encoder is... After processing by the second edge completion module, the result is... Will The input is the third decoder, the fourth encoder corresponds to the second decoder, and the output of the fourth encoder is... After processing by the second edge completion module, the result is... Will Input the second decoder, Figure 2 In the diagram, "ES1" represents the first edge supplementation module, "ES2" represents the second edge supplementation module, "ES3" represents the third edge supplementation module, and "Conv+SPI" represents the superpixel semantic primitive initialization module.

[0081] The output of the first encoder The output of the fifth encoder is used as the input to the fifth decoder. As input to the first decoder. Figure 2 The superpixel semantic primitive generation model shown uses an encoder-decoder with skip connections to extract pixel features and initialize superpixel semantic primitive features. The encoder takes a color image as input and generates a deep feature map through convolutional layers. Then, the decoder progressively upsamples the deep feature map through deconvolutional layers to obtain the final pixel features, while also considering the features of the corresponding encoder layers.

[0082] There are 5 encoders in total. Except for the first encoder, which only performs feature upscaling on the original features of the sample image, the other layers halve the width and height of the input feature map (the previous deep feature map) while increasing the dimension of the features in the channels. This allows the network to capture more extensive and abstract features at each layer while controlling the amount of computation.

[0083] There are 5 decoders in total. The first decoder layer only processes the output of the 5th encoder. Upsampling is performed, while the remaining layers fuse features input from the bottom decoder layer and side outputs input from the corresponding encoder layer or after feature transformation. The outputs of the second to the (n-1)th encoders are processed by an edge supplementation module before being input to the corresponding decoder. The edge supplementation (ES) module is used to extract and supplement edge information so that the superpixel semantic primitive generation model can have better output results.

[0084] Optionally, in the above technical solution, each edge supplementation module includes a first convolutional layer, a second convolutional layer, and a third convolutional layer, such as... Figure 3 As shown, the data processing procedure of the edge supplementation module is as follows:

[0085] S10. The received data is processed by the first convolutional layer and the second convolutional layer respectively to obtain the first feature vector and the second feature vector;

[0086] The data received by the first edge compensation module is the output of the second encoder. The data received by the second edge compensation module is the output of the third encoder. The data received by the third edge compensation module is the output of the fourth encoder. For ease of description, the data received by the edge supplementation module is labeled as Where x takes values ​​in the range of 2, 3, and 4, and is processed by the first and second convolutional layers respectively. After processing, two feature vectors are obtained, specifically the first feature vector. Second eigenvector

[0087]

[0088] in, Indicates: Through the first convolutional layer To process, Indicates: through the second convolutional layer The processing involves the first and second convolutional layers, which each include a 3×3 convolutional kernel, a batch regularization layer, and a ReLU activation layer.

[0089] S11. After performing channel attention operation on the first feature vector, multiply it by the first feature vector to obtain the third feature vector;

[0090] The first feature vector is processed by the first channel attention module. Perform channel attention operation to obtain the first feature vector. The attention weights on the channels are then combined with the first feature vector. Multiplying them together yields the third feature vector after channel attention weighting.

[0091]

[0092] in, Representation: For the first eigenvector Perform channel attention operations. Representation: For the first eigenvector Attention weights on the channels and the first feature vector Multiply the corresponding elements in the first eigenvector, that is, multiply the corresponding elements in the first eigenvector. Attention weights on the channels and the first feature vector Perform element-wise multiplication.

[0093] S12. After performing channel attention operation on the second feature vector, multiply it by the second feature vector to obtain the fourth feature vector.

[0094] The second feature vector is processed by the second channel attention module. Perform channel attention operation to obtain the second feature vector. The attention weights on the channels are then combined with the second feature vector. Multiplying them together yields the fourth feature vector after channel attention weighting.

[0095]

[0096] in, Representation: For the second eigenvector Perform channel attention operations. Representation: For the second eigenvector Attention weights on the channels and the second feature vector Multiply the corresponding elements in the second eigenvector, that is, multiply the corresponding elements in the second eigenvector. Attention weights on the channels and the second feature vector Perform element-wise multiplication.

[0097] S13. Add the first eigenvector and the third eigenvector to obtain the fifth eigenvector, specifically:

[0098] For the first eigenvector and the third eigenvector Adding them together yields the fifth feature vector with channel attention weights.

[0099]

[0100] in, Representation: For the first eigenvector and the third eigenvector Add the corresponding elements in the first eigenvector. and the third eigenvector Add the corresponding elements together.

[0101] S14. Add the second eigenvector and the fourth eigenvector to obtain the sixth eigenvector, specifically:

[0102] For the second eigenvector and the fourth eigenvector Adding them together yields a sixth feature vector with channel attention weights.

[0103]

[0104] in, Representation: For the second eigenvector and the fourth eigenvector Add the corresponding elements in the second feature vector, that is, add the corresponding elements in the second feature vector. and the fourth eigenvector Add the corresponding elements together.

[0105] S15. Perform average pooling on the fifth eigenvector to obtain the seventh eigenvector, specifically:

[0106] For the fifth eigenvector Perform average pooling to obtain the seventh feature vector.

[0107] S16. Perform max pooling on the sixth feature vector to obtain the eighth feature vector, specifically:

[0108] For the sixth eigenvector Perform max pooling to obtain the eighth feature vector.

[0109] After subtracting the seventh and eighth eigenvectors, L2 regularization is performed to obtain a feature map with one channel. This feature map with one channel represents significant edge information and is denoted as .

[0110]

[0111] in, Representation: For the fifth eigenvector Perform average pooling, that is, the seventh feature vector is... The calculated eigenvectors, Representation: For the sixth eigenvector Perform max pooling, that is, the eighth feature vector is... The calculated eigenvectors, This means: performing the difference operation on the seventh and eighth eigenvectors. Indicates: To Perform L2 regularization.

[0112] S17. Concatenate the feature map with channel 1, the fifth feature vector, and the sixth feature vector to obtain the ninth feature vector. Process the ninth feature vector through the third convolutional layer to obtain the tenth feature vector. Add the tenth feature vector to the received data to obtain the output of the edge completion module. Specifically, this is achieved through the following formula:

[0113]

[0114] in, Indicates: Output of the edge supplementation module. Indicates: will and By concatenating along the channel dimension, the ninth feature vector is obtained. The calculated eigenvectors, Indicates: through the third convolutional layer After processing, the tenth feature vector is... The calculated eigenvectors, Representation: The tenth eigenvector is compared with the first eigenvector. In addition, the third convolutional layer includes a convolution operation with a kernel size of 3×3, a batch regularization layer, and a ReLU activation layer.

[0115] Iterate through the output of the second encoder The output of the third encoder and the output of the 4th encoder That is, iterate through x=2, x=3, and x=4 to obtain the output of the first edge supplement module. Output of the second edge enhancement module and the output of the third edge supplement module

[0116] Optionally, in the above technical solution, the superpixel semantic primitive initialization module includes a fourth convolutional layer, such as... Figure 4 As shown, the data processing procedure of the superpixel semantic primitive initialization module is as follows:

[0117] S20. After processing the output of the nth encoder through the fourth convolutional layer, the eleventh feature vector is obtained, specifically:

[0118] When n=5, the output of the 5th encoder After processing by the fourth convolutional layer, the output of the fifth encoder is... Dimensionality reduction yields the eleventh eigenvector F. g The eleventh eigenvector F g The eleventh feature vector F has the same number of channels as the output of the fifth decoder. g Capable of guiding the generation of superpixel semantic primitive features, the eleventh feature vector F g The process of obtaining is specifically implemented through the following expression:

[0119]

[0120] in, Indicates: the output of the 5th encoder through the 4th convolutional layer. Dimensionality reduction processing is performed.

[0121] S21. Transform the output of the nth decoder to obtain the twelfth feature vector, specifically:

[0122] The output of the nth decoder is the pixel feature, denoted as F. p F p ∈R H×W×C Where H represents the height of the sample image, W represents the width of the training sample image, and C represents the number of channels of the pixel feature output by the nth decoder, which is also the standard number of channels in the process of calculating soft correlation. p The transformation yields the twelfth eigenvector F′. p , Among them, h num ×w num h represents the number of superpixel semantic primitives that need to be segmented. num Represents: the height of the feature map composed of superpixel semantic primitives in the initial state, w num Represents: the width h of the feature map composed of superpixel semantic primitives in the initial state. sp ×w sp Represents: the number of pixels in the pixel block contained in each superpixel semantic primitive in the initial state, h sp Represents: the height of the pixel block contained in each superpixel semantic primitive in the initial state, w sp Represents: the width of the pixel block contained in each superpixel semantic primitive in the initial state, satisfying the following condition: H = h num ×h sp

[0123] W = w num ×w sp

[0124] In actual model training, H and h sp Both are fixed values, W and w sp All are fixed values. S22. Transform the eleventh eigenvector to obtain the thirteenth eigenvector. Wherein, the eleventh eigenvector... F g The transformation yields the thirteenth eigenvector F′. g , S23. Based on the twelfth and thirteenth feature vectors, the superpixel semantic primitive features are obtained, specifically through the following formula:

[0125]

[0126] Among them, F sp Representation: Superpixel semantic primitive features, Linear1(F′) g This indicates that the thirteenth feature vector F′ is processed through the first linear layer. g To process, Representation: Through the second linear layer, the twelfth feature vector F′ p To process, T represents matrix transpose. Indicates: To Perform a SoftMax operation. Optionally, the specific implementation process of S4 is as follows: Based on the superpixel semantic primitive features of the target image, a soft association relationship between the pixels of the target image and the superpixel semantic primitives of the target image is established using a differentiable soft clustering method. Specifically, the soft association relationship between the pixels of the target image and the superpixel semantic primitives of the target image is established by calculating the similarity between each pixel of the target image and its nine surrounding superpixel semantic primitives. This soft association relationship is specifically represented by the allocation matrix Q.

[0127]

[0128]

[0129] Among them, F′ p This means that after inputting the target image into the trained superpixel semantic primitive generation model, the transformed feature vector of the output of the nth decoder of the trained superpixel semantic primitive generation model can be simply considered as F′. p F′ is the twelfth feature vector corresponding to the target image. sp Representation: The feature vector obtained after transforming the superpixel semantic primitive features of the target image. This indicates that each superpixel semantic primitive contains the features of its nine surrounding superpixel semantic primitives, including itself. 2 The symbol represents the operation of exponentiation to the square, ⊙ represents multiplication at corresponding positions, and Sgn(x) is a sign function used to preserve the sign of the input parameter x. x is the input parameter, and the sign function returns the sign of the input parameter x. Sgn(F′) p ·(F′ sp ) T ) indicates that F′ is retained. p ·(F′ sp ) T The calculated result is positive or negative (because the latter half of the allocation matrix Q involves a squaring operation, resulting in the loss of F′). p ·(F′ sp ) T (The sign of the calculated result).

[0130] In this method, the specific steps for extracting features from a given image using a traditional convolutional neural network and obtaining image features at different extraction stages include:

[0131] The input is an RGB color image. A five-stage small "U"-shaped network is constructed with channel dimensions of [16, 32, 64, 128, 256], respectively. The deepest layer features are extracted as seed guiding features for superpixel clustering centers, and the final decoder output is used as the feature of the pixels to be clustered.

[0132] Optionally, in the above technical solution, the specific steps of using a uniquely designed edge extractor to extract edges from given shallow convolutional features and obtain edge enhancements for the convolutional network include:

[0133] The input feature map is passed through two different 3x3 convolutional layers to obtain two feature maps. Then, each feature map is passed through a channel attention layer to select the dimension containing edge information. After passing through an average pooling layer and a max pooling layer, the difference between the two results is calculated and L2 regularization is applied to the channel dimension to obtain the boundary map. The boundary map is then upsampled, concatenated with the two feature maps, and passed back to the original input feature map dimension through a 1x1 convolution. The residual is then connected to the original input feature map.

[0134] Optionally, in the above technical solution, a uniquely designed attention mechanism is used to guide the aggregation of the neural network's output features through the deepest layer features to obtain the initial superpixel semantic primitive seed. Specific steps include:

[0135] First, this part includes two dimension reduction layers. A 1×1 convolutional kernel is used on the superpixel semantic primitive seed features, supplemented by batch normalization (BN) and ReLU activation function, so that its dimension is the same as the pixel feature dimension.

[0136] The pixel features are divided into small squares with a stride of 16, at which point the number of squares is the same as the number of superpixel semantic primitive seed features. Each attention module receives a superpixel semantic primitive seed feature and the corresponding 16x16 pixel feature as input, and uses the seed feature to select and aggregate pixel features.

[0137] An improved similarity evaluation method is used to generate superpixel semantic primitive seeds and pixel information similarity metric matrices, resulting in the final soft assignment matrix. Specific steps include:

[0138] Pixel features are multiplied and summed with the corresponding channels of the nine adjacent superpixel semantic primitive seeds. The result is squared and multiplied by a sign function (Sgn), which serves as the basis for similarity measurement, generating a similarity matrix with 9 channels. A softmax normalization operation is then performed along the channel dimension, assigning pixels as a percentage to the surrounding nine superpixel semantic primitives. This same operation is performed on all pixels to obtain a superpixel soft-assignment matrix based on a modified similarity metric, distance.

[0139] Optionally, in the above technical solution, the object label information, pixel coordinate information, RGB color information of the semantic segmentation dataset are generated and reconstructed using a soft allocation matrix, and the corresponding information is used to construct a cross-entropy loss optimization.

[0140] Optionally, in the above technical solution, the selected optimization method is the Adaptive Moment Estimation Algorithm (Adam) in the deep learning framework (PyTorch).

[0141] In this invention, a traditional convolutional neural network is used to extract features from a given image, resulting in image features at different extraction stages. A uniquely designed edge extractor is used to extract edges from given shallow convolutional features, enhancing the edge information of the convolutional features. Through a uniquely designed attention mechanism, the deepest layer features of the neural network are used to guide the aggregation of the neural network's output features, obtaining the initial superpixel semantic primitive seed. An improved similarity evaluation method is used to generate the superpixel semantic primitive seed and a pixel information similarity metric matrix, resulting in the final soft-assignment matrix, which is then used to generate semantic primitives.

[0142] This invention constructs a soft clustering superpixel semantic primitive generation model based on an edge enhancement strategy to perform superpixel segmentation on a given arbitrary color image and generate features for superpixel semantic primitives. To this end, the invention first uses a 5-stage "U"-shaped network structure for image feature extraction, where the input end (left) is an encoder and the output end (right) is a decoder. Corresponding layers on both sides of the network are skipped, and this small convolutional neural network extracts features from the image. Specifically, firstly, for shallow image features, unlike previous methods, we extract object boundary information from shallow features, which can preserve the structural information of complex images to a greater extent, making the segmentation boundaries more accurate and clear. Secondly, this invention constructs a deep convolutional feature to guide local feature aggregation, thereby adaptively obtaining initial soft cluster centers. This aims to generate an ideal superpixel allocation matrix with fewer iterations. Simultaneously, the allocation matrix is ​​used to generate superpixel semantic primitives, avoiding the unacceptable time consumption caused by multiple iterations of clustering, thus improving the overall performance in the superpixel semantic primitive generation task. Finally, this invention improves the similarity measurement method by using a modified dot product distance calculation for similarity evaluation, further strengthening the correlation between similar features and suppressing the interference of low-correlation features on the soft clustering results, thus obtaining a more accurate allocation matrix. The shallow feature boundary extractor constructed in this invention can extract clear object boundary information. Comprehensive experiments demonstrate that the edge information enhancement module of this invention can effectively improve the representation ability of pixel features.

[0143] This invention proposes a novel attention-enhanced superpixel semantic primitive feature initialization construction module, capable of progressively and adaptively calculating superpixel semantic primitive seed features. We innovatively utilize the deepest convolutional features rich in global information, inputting them as guiding information for superpixel semantic primitive features into the attention module to guide pixel feature aggregation and the construction of superpixel semantic primitive seed features. Experiments demonstrate that the model provided by this invention possesses stronger superpixel semantic primitive seed extraction capabilities, achieving good superpixel semantic primitive generation results with fewer iterations.

[0144] The final pixel allocation matrix of the model proposed in this invention is obtained by an improved vector dot product soft clustering method. The module functionality will not completely fail due to changes in the task being addressed. That is, after being integrated into a multimodal document processing framework, this model can continue to be trained together with it, ensuring sufficiently good superpixel semantic primitive generation results. Therefore, after we have pre-trained the model, this model can be directly embedded into a multimodal document processing framework without explicit constraints.

[0145] The superpixel semantic primitive generation model in this invention, as well as existing models such as LNSNet, FCN, Alnet, and SLIC, are trained using an international standard dataset. The segmentation performance of the trained models is evaluated using multiple evaluation metrics. The international standard dataset can be the BSDS500 dataset, the source of which is as follows: Figure 5 As shown.

[0146] The existing LNSNet, FCN, Alnet, and SLIC models originate from, for example... Figure 6 As shown.

[0147] Several evaluation metrics include: superpixel count and edge recall. Figure 7 The data shows a comparison of the number of superpixels in the trained models. Figure 8 The data shows a comparison of the boundary recall rates of the trained models. Figure 9 The visualization results of the outputs of multiple trained models are shown. Figure 7 , Figure 8 and Figure 9 In this context, "ours" refers to the trained superpixel semantic primitive generation model in this invention. Although the steps are numbered S1, S2, etc., in the above embodiments, these are merely specific embodiments provided by this invention. Those skilled in the art can adjust the execution order of S1, S2, etc., according to actual circumstances, which is also within the scope of protection of this invention. It is understood that in some embodiments, some or all of the above embodiments may be included.

[0148] like Figure 10 As shown, a superpixel semantic primitive generation system 200 according to an embodiment of the present invention includes a model building module 201, a model training module 202, a target image processing module 203, a soft association relationship determination module 204, and a superpixel semantic primitive generation module 205.

[0149] Model building module 201 is used to: build a superpixel semantic primitive generation model;

[0150] Model training module 202 is used to: train the superpixel semantic primitive generation model to obtain the trained superpixel semantic primitive generation model;

[0151] The target image processing module 203 is used to: process the target image using the trained superpixel semantic primitive generation model to obtain the superpixel semantic primitive features of the target image;

[0152] The soft association determination module 204 is used to: establish soft association relationships between pixels of the target image and superpixel semantic primitives of the target image based on the superpixel semantic primitive features of the target image;

[0153] The superpixel semantic primitive generation module 205 is used to: obtain the superpixel semantic primitives of the target image based on the pixels of the target image and the soft correlation relationship.

[0154] Optionally, in the above technical solution, the superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged in sequence, n decoders arranged in sequence, and n-2 edge supplementation modules; the first encoder is used to receive sample images, the n encoders and n decoders correspond one-to-one, in two adjacent encoders, the output of the previous encoder is used as the input of the next encoder, in two adjacent decoders, the output of the previous decoder is used as the input of the decoder, the output of the first encoder is used as the input of the nth decoder, the output of the nth encoder is used as the input of the first decoder, and the output of the second encoder to the (n-1)th encoder is processed by an edge supplementation module before being input to the corresponding decoder;

[0155] The outputs of the nth encoder and the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image.

[0156] Optionally, in the above technical solution, the data processing procedure of the edge supplementation module is as follows:

[0157] The received data is processed by the first and second convolutional layers to obtain the first and second feature vectors, respectively. The first feature vector is then multiplied by the first feature vector after channel attention, resulting in the third feature vector. The second feature vector is then multiplied by the second feature vector to obtain the fourth feature vector. The first and third feature vectors are added together to obtain the fifth feature vector. The second and fourth feature vectors are added together to obtain the sixth feature vector. The fifth feature vector is then subjected to average pooling to obtain the seventh feature vector. The sixth feature vector is then subjected to max pooling to obtain the eighth feature vector. The difference between the seventh and eighth feature vectors is then calculated, followed by L2 regularization to obtain a feature map with one channel. The feature map with one channel, the fifth feature vector, and the sixth feature vector are concatenated to obtain the ninth feature vector. The ninth feature vector is then processed by the third convolutional layer to obtain the tenth feature vector. Finally, the tenth feature vector is added to the received data to obtain the output of the edge completion module.

[0158] Optionally, in the above technical solution, the data processing procedure of the superpixel semantic primitive initialization module is as follows:

[0159] The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector; the output of the nth decoder is transformed to obtain the twelfth feature vector; the eleventh feature vector is transformed to obtain the thirteenth feature vector; the superpixel semantic primitive features are obtained based on the twelfth and thirteenth feature vectors.

[0160] It should be noted that the beneficial effects of the superpixel semantic primitive generation system 200 provided in the above embodiments are the same as those of the superpixel semantic primitive generation method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0161] The superpixel semantic primitive generation system of the present invention can be a computer program (including program code) running on a computer device. For example, the superpixel semantic primitive generation system of the present invention is an application software that can be used to execute the corresponding steps in the superpixel semantic primitive generation method of the present invention.

[0162] In some embodiments, the superpixel semantic primitive generation system of the present invention can be implemented in a combination of hardware and software. As an example, the superpixel semantic primitive generation system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the superpixel semantic primitive generation method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0163] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0164] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the superpixel semantic primitive generation methods described above. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the superpixel semantic primitive generation method shown in any embodiment of the present invention by calling the computer program.

[0165] In one alternative embodiment, an electronic device is provided, such as Figure 11 As shown, Figure 11 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0166] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0167] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0168] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0169] The memory 4003 stores the application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0170] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0171] It should be noted that, Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0172] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described superpixel semantic primitive generation methods.

[0173] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0174] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the aforementioned superpixel semantic primitive generation.

[0175] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0176] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0177] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0178] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0179] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0180] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0181] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0182] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for generating superpixel semantic primitives, characterized in that, include: Construct a superpixel semantic primitive generation model; The superpixel semantic primitive generation model is trained to obtain a trained superpixel semantic primitive generation model. The target image is processed using a trained superpixel semantic primitive generation model to obtain the superpixel semantic primitive features of the target image; Based on the superpixel semantic primitive features of the target image, a soft association relationship is established between the pixels of the target image and the superpixel semantic primitives of the target image; Based on the pixels of the target image and the soft association relationship, the superpixel semantic primitives of the target image are obtained; Specifically, establishing a soft association relationship between pixels of the target image and its superpixel semantic primitives based on the superpixel semantic primitive features of the target image includes: A soft association relationship between pixels and superpixel semantic primitives of the target image is established by calculating the similarity between each pixel of the target image and its nine surrounding superpixel semantic primitives. This soft association relationship is specifically represented by the assignment matrix Q. Among them, F′ p Representation: After inputting the target image into the trained superpixel semantic primitive generation model, the transformed feature vector F′ is obtained by processing the output of the nth decoder of the trained superpixel semantic primitive generation model. sp Representation: The feature vector obtained after transforming the superpixel semantic primitive features of the target image. This indicates that each superpixel semantic primitive contains the features of its nine surrounding superpixel semantic primitives, including itself. 2 The symbol represents the operation of exponentiation to the square, ⊙ represents multiplication at corresponding positions, and Sgn(x) is a sign function used to preserve the sign of the input parameter x. x is the input parameter, and the sign function returns the sign of the input parameter x. Sgn(F′) p ·(F′ sp ) T ) indicates that F′ is retained. p ·(F′ sp ) T The calculated result is positive or negative; The superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged sequentially, n decoders arranged sequentially, and n-2 edge completion modules. The first encoder is used to receive the sample image. The n encoders and n decoders correspond one-to-one. In two adjacent encoders, the output of the previous encoder is used as the input of the next encoder. In two adjacent decoders, the output of the previous decoder is used as the input of the decoder. The output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder. The outputs of the second to (n-1)th encoders are processed by an edge completion module before being input to the corresponding decoder. The outputs of the nth encoder and the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image. The data processing procedure of the edge compensation module is as follows: The received data is processed by the first convolutional layer and the second convolutional layer respectively to obtain the first feature vector and the second feature vector; After performing channel attention operation on the first feature vector, it is multiplied with the first feature vector to obtain the third feature vector; After performing channel attention operation on the second feature vector, it is multiplied with the second feature vector to obtain the fourth feature vector; The first feature vector and the third feature vector are added together to obtain the fifth feature vector; Add the second feature vector and the fourth feature vector to obtain the sixth feature vector; The fifth feature vector is subjected to average pooling to obtain the seventh feature vector; Perform max pooling on the sixth feature vector to obtain the eighth feature vector; After performing the difference operation on the seventh feature vector and the eighth feature vector, L2 regularization is then performed to obtain a feature map with 1 channel. The feature map with channel 1, the fifth feature vector, and the sixth feature vector are concatenated to obtain the ninth feature vector. The ninth feature vector is then processed by the third convolutional layer to obtain the tenth feature vector. The tenth feature vector is then added to the received data to obtain the output of the edge supplementation module. The data processing procedure of the superpixel semantic primitive initialization module is as follows: The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector; Transform the output of the nth decoder to obtain the twelfth feature vector; The eleventh feature vector is transformed to obtain the thirteenth feature vector; The superpixel semantic primitive features are obtained based on the twelfth and thirteenth feature vectors; The output of the nth decoder is transformed to obtain the twelfth feature vector, which includes: The output of the nth decoder is the pixel feature, denoted as F. p F p ∈R H×W×C Where H represents the height of the sample image, W represents the width of the training sample image, and C represents the number of channels of the pixel feature output by the nth decoder, which is also the standard number of channels in the process of calculating soft correlation. p The transformation yields the twelfth eigenvector F′. p , Among them, h num ×w num h represents the number of superpixel semantic primitives that need to be segmented. num Represents: the height of the feature map composed of superpixel semantic primitives in the initial state, w num Represents: the width h of the feature map composed of superpixel semantic primitives in the initial state. sp ×w sp Represents: the number of pixels in the pixel block contained in each superpixel semantic primitive in the initial state, h sp Represents: the height of the pixel block contained in each superpixel semantic primitive in the initial state, w sp Represents: the width of the pixel block contained in each superpixel semantic primitive in the initial state, satisfying the following condition: H=h num ×h sp W=w num ×w sp The superpixel semantic primitive features are obtained based on the twelfth and thirteenth feature vectors, and are achieved through the following formula: Among them, F sp Representation: Superpixel semantic primitive features, Linear1(F′) g This indicates that the thirteenth feature vector F′ is processed through the first linear layer. g To process, Representation: Through the second linear layer, the twelfth feature vector F′ p To process, T represents matrix transpose. Indicates: To Perform a SoftMax operation.

2. A superpixel semantic primitive generation system, characterized in that, It includes a model building module, a model training module, a target image processing module, a soft relation determination module, and a superpixel semantic primitive generation module; The model building module is used to: build a superpixel semantic primitive generation model; The model training module is used to: train the superpixel semantic primitive generation model to obtain a trained superpixel semantic primitive generation model; The target image processing module is used to: process the target image using a trained superpixel semantic primitive generation model to obtain the superpixel semantic primitive features of the target image; The soft association relationship determination module is used to: establish a soft association relationship between the pixels of the target image and the superpixel semantic primitives of the target image based on the superpixel semantic primitive features of the target image; The superpixel semantic primitive generation module is used to: obtain the superpixel semantic primitives of the target image based on the pixels of the target image and the soft correlation relationship; Specifically, establishing a soft association relationship between pixels of the target image and its superpixel semantic primitives based on the superpixel semantic primitive features of the target image includes: A soft association relationship between pixels and superpixel semantic primitives of the target image is established by calculating the similarity between each pixel of the target image and its nine surrounding superpixel semantic primitives. This soft association relationship is specifically represented by the assignment matrix Q. Among them, F′ p Representation: After inputting the target image into the trained superpixel semantic primitive generation model, the transformed feature vector F′ is obtained by processing the output of the nth decoder of the trained superpixel semantic primitive generation model. sp Representation: The feature vector obtained after transforming the superpixel semantic primitive features of the target image. This indicates that each superpixel semantic primitive contains the features of its nine surrounding superpixel semantic primitives, including itself. 2 The symbol represents the operation of exponentiation to the square, ⊙ represents multiplication at corresponding positions, and Sgn(x) is a sign function used to preserve the sign of the input parameter x. x is the input parameter, and the sign function returns the sign of the input parameter x. Sgn(F′) p ·(F′ sp ) T ) indicates that F′ is retained. p ·(F′ sp ) T The calculated result is positive or negative; The superpixel semantic primitive generation model includes: a superpixel semantic primitive initialization module, n encoders arranged sequentially, n decoders arranged sequentially, and n-2 edge completion modules. The first encoder is used to receive the sample image. The n encoders and n decoders correspond one-to-one. In two adjacent encoders, the output of the previous encoder is used as the input of the next encoder. In two adjacent decoders, the output of the previous decoder is used as the input of the decoder. The output of the first encoder is used as the input of the nth decoder, and the output of the nth encoder is used as the input of the first decoder. The outputs of the second to (n-1)th encoders are processed by an edge completion module before being input to the corresponding decoder. The outputs of the nth encoder and the nth decoder are processed by the superpixel semantic primitive initialization module to obtain the superpixel semantic primitive features of the sample image. The data processing procedure of the edge compensation module is as follows: The received data is processed by the first convolutional layer and the second convolutional layer respectively to obtain the first feature vector and the second feature vector; After performing channel attention operation on the first feature vector, it is multiplied with the first feature vector to obtain the third feature vector; After performing channel attention operation on the second feature vector, it is multiplied with the second feature vector to obtain the fourth feature vector; The first feature vector and the third feature vector are added together to obtain the fifth feature vector; Add the second feature vector and the fourth feature vector to obtain the sixth feature vector; The fifth feature vector is subjected to average pooling to obtain the seventh feature vector; Perform max pooling on the sixth feature vector to obtain the eighth feature vector; After performing the difference operation on the seventh feature vector and the eighth feature vector, L2 regularization is then performed to obtain a feature map with 1 channel. The feature map with channel 1, the fifth feature vector, and the sixth feature vector are concatenated to obtain the ninth feature vector. The ninth feature vector is then processed by the third convolutional layer to obtain the tenth feature vector. The tenth feature vector is then added to the received data to obtain the output of the edge supplementation module. The data processing procedure of the superpixel semantic primitive initialization module is as follows: The output of the nth encoder is processed by the fourth convolutional layer to obtain the eleventh feature vector; Transform the output of the nth decoder to obtain the twelfth feature vector; The eleventh feature vector is transformed to obtain the thirteenth feature vector; The superpixel semantic primitive features are obtained based on the twelfth and thirteenth feature vectors; The output of the nth decoder is transformed to obtain the twelfth feature vector, which includes: The output of the nth decoder is the pixel feature, denoted as F. p F p ∈R H×W×C Where H represents the height of the sample image, W represents the width of the training sample image, and C represents the number of channels of the pixel feature output by the nth decoder, which is also the standard number of channels in the process of calculating soft correlation. p The transformation yields the twelfth eigenvector F. p ′ , Among them, h num ×w num h represents the number of superpixel semantic primitives that need to be segmented. num Represents: the height of the feature map composed of superpixel semantic primitives in the initial state, w num Represents: the width h of the feature map composed of superpixel semantic primitives in the initial state. sp ×w sp Represents: the number of pixels in the pixel block contained in each superpixel semantic primitive in the initial state, h sp Represents: the height of the pixel block contained in each superpixel semantic primitive in the initial state, w sp Represents: the width of the pixel block contained in each superpixel semantic primitive in the initial state, satisfying the following condition: H=h num ×h sp W=w num ×w sp The superpixel semantic primitive features are obtained based on the twelfth and thirteenth feature vectors, and are achieved through the following formula: Among them, F sp Representation: Superpixel semantic primitive features, Linear1(F′) g This indicates that the thirteenth feature vector F′ is processed through the first linear layer. g To process, Representation: Through the second linear layer, the twelfth feature vector F′ p To process, T represents matrix transpose. Indicates: To Perform a SoftMax operation.

3. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the superpixel semantic primitive generation method of claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the superpixel semantic primitive generation method of claim 1.

Images