Open World Semantic Segmentation Model Construction Method

Abstract

The embodiment of the present disclosure discloses a method for constructing an open-world semantic segmentation model. A specific implementation of the method includes: generating various mixed semantic image data; determining a mixed semantic image data set; obtaining various depth feature information and various first energy score matrices; obtaining various pixel-level feature information, various first unknown semantic scores, various self-attention maps, and various second energy score matrices; generating various calibration pixel-level feature similarities; obtaining various calibration pixel-level feature information and various first unknown semantic calibration scores; obtaining semantically relevant feature information, second unknown semantic scores, and semantically irrelevant feature information; generating various overlapping area semantic feature information; obtaining various reconstructed image data; generating a target loss value; and determining the open-world semantic segmentation network as an open-world semantic segmentation model. This implementation can train a more reliable open-world semantic segmentation model and improve the recognition accuracy of the model.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a method for constructing an open world semantic segmentation model. Background Art

[0002] Accurate semantic segmentation is crucial for many applications, including autonomous driving, medical image analysis, and intelligent surveillance. Only by precisely segmenting distinct semantic regions within an image can autonomous driving accurately identify and avoid obstacles, and accurately identify and label lesions in medical images. However, traditional image processing methods often rely on handcrafted features and rules, making them difficult to adapt to complex and changing scenarios. Deep learning-based semantic segmentation methods, by learning from large amounts of data, can automatically extract rich feature information, resulting in greater robustness and adaptability. With the advancement of deep learning, particularly convolutional neural networks, semantic segmentation technology has achieved significant progress in accuracy and efficiency. Compared to other methods, deep learning-based semantic segmentation not only offers significant improvements in accuracy but also boasts advantages such as end-to-end training, high automation, and strong adaptability. Therefore, it holds broad application potential and significant research value in areas such as autonomous driving, medical imaging, and intelligent surveillance.

[0003] Existing semantic segmentation technologies typically train models on closed-set data (within the distribution). These models can only recognize categories defined within the distribution and cannot recognize categories that have not been seen during training, that is, unknown semantic (out-of-distribution) inputs. This causes the model to identify unknown categories as known categories when faced with unknown inputs, posing a huge safety hazard. For example, a semantic segmentation model trained on a closed-set street view dataset cannot recognize unknown obstacles in an open scene, posing a serious threat to the safety of self-driving cars. Therefore, it is crucial to accurately identify pixels of unknown semantic categories while accurately segmenting pixels of within-distribution categories. This can be defined as: open-world semantic segmentation, that is, based on known categories, effective mechanisms or methods are used to detect and distinguish unknown semantic categories to improve the safety and reliability of the model in practical applications.

[0004] Pioneers have conducted varying degrees of research on open-world semantic segmentation. W. Liu et al. estimated the uncertainty of neural network outputs to determine whether each pixel is an unknown semantic pixel, using metrics such as energy score, Euclidean distance, and entropy to assess uncertainty. This approach does not require retraining the neural network, but it cannot effectively distinguish difficult samples that are similar to those in the distribution. Y. Xia et al. reconstructed the features or semantic map output by the neural network and compared the reconstruction with the original image. Pixels with large reconstruction errors were considered unknown semantic pixels. Because the reconstructed network overfits the appearance of in-distribution data, this method is difficult to generalize to scenarios with large domain shifts and incurs significant additional computational overhead. Y. Liu et al. introduced an additional dataset as unknown data, cut-and-pasted it onto the in-distribution dataset, and added modules to a closed-set pre-trained model to fit the unknown data, thus enabling the model to detect open-set semantics. Although this approach has good unknown semantic recognition capabilities, it requires a large number of additional annotated datasets, which are often labor-intensive and difficult to obtain.

[0005] Technical Issue: Existing semantic segmentation technologies typically train models on known data (in-distribution). These models can only recognize categories defined by the known data and are unable to recognize data not seen during training, i.e., unknown semantic data (out-of-distribution). This causes the model to misidentify unknown semantic data as known data when faced with input, resulting in low reliability and accuracy of model recognition. Summary of the Invention

[0006] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] Some embodiments of the present disclosure propose a method for constructing an open-world semantic segmentation model to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a method for constructing an open-world semantic segmentation model, the method comprising: generating each mixed semantic image data based on each preset known semantic image data; determining a preset number of mixed semantic image data in the above-mentioned each mixed semantic image data as a mixed semantic image dataset; performing the following training steps based on the mixed semantic image dataset: inputting the mixed semantic image dataset into a semantic segmentation network in an open-world semantic segmentation network to obtain each deep feature information and each first energy score matrix; inputting the above-mentioned each deep feature information into a micro-decoder in the open-world semantic segmentation network to decode the above-mentioned each mixed semantic image data to obtain each pixel-level feature information, each first unknown semantic score, each self-attention map and each second energy score matrix; generating each calibrated pixel-level feature similarity based on the above-mentioned each self-attention map; calibrating the above-mentioned each pixel-level feature information and the above-mentioned each first unknown semantic score based on the above-mentioned each calibrated pixel-level feature similarity to obtain each calibrated pixel-level feature information and each first unknown semantic calibration score; for each calibrated pixel-level feature information in the above-mentioned each calibrated pixel-level feature information, inputting the above-mentioned calibrated pixel-level feature information into the open-world semantic segmentation network. The semantically relevant decoupler and the semantically irrelevant decoupler in the training process are used to perform feature decoupling on the calibrated pixel-level feature information to obtain semantically relevant feature information, second unknown semantic scores, and semantically irrelevant feature information; based on the obtained semantically relevant feature information and the obtained semantically irrelevant feature information, semantic feature information of each overlapping area is generated; image reconstruction processing is performed on the semantic feature information of each overlapping area to obtain reconstructed image data; based on the first energy score matrices, the second energy score matrices, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data, and the semantically irrelevant feature information, a target loss value is generated; in response to determining that the target loss value meets the preset loss value condition, the open-world semantic segmentation network is determined as the open-world semantic segmentation model, and the first unknown semantic calibration scores obtained in the last training step are determined as the target calibration scores; in response to determining that the target loss value does not meet the preset loss value condition, the network parameters of the open-world semantic segmentation network are adjusted, and unused mixed semantic image data are used to form a mixed semantic image dataset, and the adjusted open-world semantic segmentation network is used as the open-world semantic segmentation network to perform the training step again.

[0009] The above embodiments of the present disclosure have the following beneficial effects:

[0010] 1. The present invention discloses an open-world semantic segmentation algorithm. Without introducing additional data, it uses in-distribution data to synthesize a large amount of unknown semantic data, alleviating the problem of missing unknown semantic data. Through the self-supervised training method of "decoupling + reconstruction", it alleviates the problem of overfitting of the model to the generated data, greatly improving the ability of the semantic segmentation model to recognize unknown inputs.

[0011] 2. The present invention proposes a dynamic penalty loss function, which dynamically penalizes the model by estimating the possibility that pixels have unknown semantics, thereby alleviating the label noise introduced by synthetic data. At the same time, the present invention proposes an inter-pixel feature similarity calibration module, which calibrates inaccurate inter-pixel feature similarities through a hybrid self-attention approach, clarifies the decision boundaries between distributions, and improves the model's ability to identify pixels with unknown semantics.

[0012] 3. The method proposed in this invention can be migrated to deep learning semantic segmentation models of various architectures, and has the characteristics of simple design, high robustness, and high detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flowchart of some embodiments of the method for building an open-world semantic segmentation model according to the present disclosure;

[0015] Figure 2 A schematic diagram of generating various mixed semantic image data based on various known semantic image data according to the open world semantic segmentation model construction method of some embodiments of the present disclosure;

[0016] Figure 3 A schematic diagram of generating calibrated pixel-level feature similarity, calibrated pixel-level feature information, and a first unknown semantic calibration score according to an open-world semantic segmentation model construction method according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 A process 100 of some embodiments of the method for building an open-world semantic segmentation model according to the present disclosure is shown. The method for building an open-world semantic segmentation model includes the following steps:

[0024] Step 101: Generate mixed semantic image data based on preset known semantic image data.

[0025] In some embodiments, various mixed semantic image data can be generated based on various preset known semantic image data. The known semantic image data in the aforementioned various known semantic image data may be image data in which each pixel point is a known semantic pixel point. The aforementioned known semantic pixel point may be a pixel point whose corresponding semantic label is known. The aforementioned semantic label may be a label for characterizing the category corresponding to the pixel point. The mixed semantic image data in the aforementioned various mixed semantic image data may be image data containing known semantic pixel points and unknown semantic pixel points. The aforementioned unknown semantic pixel point may be a pixel point whose corresponding semantic label is unknown.

[0026] In some optional implementations of some embodiments, various mixed semantic image data may be generated based on various preset known semantic image data through the following steps:

[0027] In the first step, a known semantic image data is randomly selected from each known semantic image data acquired in advance as a known semantic sample image.

[0028] The second step is to preprocess the known semantic sample image to obtain a preprocessed sample image. In practice, the known semantic sample image can be randomly scaled and then color-enhanced using image enhancement techniques to obtain the preprocessed sample image. The image enhancement technique can be false color processing.

[0029] The third step is to crop the pre-processed sample image based on the preset shape information to obtain image patches. The shape information in the shape information can be used to characterize the shape and size of the cropped image when the pre-processed sample image is cropped. For example, the shape information can be a square image with a size of 20*20. The image patch in the image patches can be the image area in the pre-processed sample image. In practice, for each shape information in the shape information, the pre-processed sample image can be randomly cropped into the shape and size corresponding to the shape information through the image processing library to obtain an image patch. The image processing library can be the OpenCV library.

[0030] The fourth step is to randomly paste each of the above image patches onto a preset sample image to obtain a patch sample image. The sample image may be any known semantic image data other than the known semantic sample image. The patch sample image may be a sample image onto which each image patch has been pasted. In practice, the patch sample image may be obtained by randomly pasting each of the above image patches onto the sample image using an image processing library. The image processing library may be the OpenCV library.

[0031] Optionally, before randomly pasting the image patches onto a preset sample image to obtain a patch sample image, the sample image may be randomly scaled to obtain a randomly scaled sample image as the sample image to update the sample image. Subsequently, the image patches may be randomly pasted onto the updated sample image to obtain a patch sample image.

[0032] The fifth step is to crop the patch sample image based on the preset cropping information to obtain individual cropped sample images. The cropping information may be information for characterizing the method used to crop the patch sample image. For example, the cropping information may be "cropping using a preset number of rectangles with overlapping parts". The cropped sample images in the individual cropped sample images may be images obtained by cropping the patch sample image. Every two cropped sample images in the individual cropped sample images have the same image area. In practice, the patch sample image may be cropped according to the method corresponding to the cropping information through a library function to obtain individual cropped sample images corresponding to the patch sample image. The library function may be crop().

[0033] The sixth step is to randomly mask each of the above-mentioned cropped sample images to obtain each masked sample image. Among them, the masked sample image in each of the above-mentioned masked sample images can be an image obtained after randomly masking the cropped sample image. Every two masked sample images in each of the above-mentioned masked sample images have the same image area. In practice, for each of the above-mentioned cropped sample images, the pixel values ​​corresponding to the preset number of pixels in the above-mentioned cropped sample image can be set to the preset pixel value through the above-mentioned image processing library to obtain a masked sample image. The above-mentioned preset number of pixels can be a preset numerical value. Here, there is no limitation on the specific setting of the above-mentioned preset number of pixels. The above-mentioned preset pixel value can be a numerical value between 0 and 255. Here, there is no limitation on the specific setting of the above-mentioned preset pixel value.

[0034] In the seventh step, the image patches that meet a preset shape condition among the image patches are determined as target image patches, wherein the preset shape condition may be that the shapes of the image patches are the same.

[0035] In the eighth step, for each of the mask sample images above, the target image patches that meet the preset pasting conditions among the target image patches are pasted to the preset positions in the mask sample images above to obtain mixed semantic image data. The preset positions may be any image region in the mask sample image that is the same as the target mask sample images. The target mask sample images may be mask sample images other than the mask sample images among the mask sample images above. In practice, for each of the mask sample images above, the target image patches that meet the preset pasting conditions among the target image patches may be pasted to the preset positions in the mask sample images above through the image processing library to obtain mixed semantic image data.

[0036] Optionally, after pasting the target image patches that meet the preset pasting conditions in the above-mentioned target image patches to the preset positions in the above-mentioned mask sample images, the mask sample images after pasting the target image patches can also be subjected to brightness adjustment, saturation adjustment, hue adjustment, etc. using data enhancement technology to obtain respective adjusted mask sample images. The adjusted mask sample images in the above-mentioned adjusted mask sample images can be mask sample images that have undergone brightness adjustment, saturation adjustment, hue adjustment, etc. Then, the above-mentioned adjusted mask sample images can be determined as respective mixed semantic image data.

[0037] Step 102 : Determine a preset number of mixed semantic image data in each mixed semantic image data as a mixed semantic image data set.

[0038] In some embodiments, a preset number of mixed semantic image data in each of the mixed semantic image data can be determined as a mixed semantic image dataset. The preset number of images can be a pre-set value. Here, the specific setting of the preset number of images is not limited.

[0039] Step 103: Based on the mixed semantic image dataset, perform the following training steps:

[0040] Step 1031 : Input the mixed semantic image dataset into a semantic segmentation network in an open-world semantic segmentation network to obtain various depth feature information and various first energy score matrices.

[0041] In some embodiments, a mixed semantic image dataset can be input into a semantic segmentation network in an open-world semantic segmentation network to obtain various depth feature information and various first energy score matrices. The depth feature information in each of the depth feature information can be a feature vector corresponding to the mixed semantic image data. In each of the first energy score matrices, each row can correspond to a row of pixels in the mixed semantic image data, each column can correspond to a column of pixels in the mixed semantic image data, and each element can be a matrix containing semantic label distribution data corresponding to each pixel in the mixed semantic image data. The semantic label distribution data can be data representing the probability of a pixel belonging to each semantic label. For example, the semantic label distribution data can be "semantic label: sky, probability: 90%; semantic label: earth, probability: 10%." In practice, the mixed semantic image dataset can be input into an encoder in a semantic segmentation network to obtain multi-scale features corresponding to each mixed semantic image data in the mixed semantic image dataset. The obtained multi-scale features can then be input into a decoder in the semantic segmentation network to obtain depth feature information corresponding to each mixed semantic image data in the mixed semantic image dataset. Finally, the obtained depth feature information can be input into the classification head in the above semantic segmentation network to obtain the first energy score matrix corresponding to each mixed semantic image data in the above mixed semantic image dataset. The above open world semantic segmentation network can include four layers.

[0042] The first layer may be a semantic segmentation network. The semantic segmentation network may include an encoder, a decoder, and a classification head. The encoder may be a neural network that takes the mixed semantic image data as input and outputs multi-scale features corresponding to the mixed semantic image data. For example, the encoder may be a convolutional neural network. The decoder may be a neural network that takes the multi-scale features corresponding to the mixed semantic image data as input and outputs depth feature information corresponding to the mixed semantic image data. For example, the decoder may be a convolutional neural network. The classification head may be a linear layer that takes the depth feature information corresponding to the mixed semantic image data as input and outputs a first energy score matrix corresponding to the mixed semantic image data.

[0043] The second layer may be a micro-decoder. The micro-decoder may include a decoding module, a classification head, and a calibration module. The decoding module may be a convolutional layer composed of three layers of convolutional blocks, which takes the depth feature information corresponding to the mixed semantic image data as input and the pixel-level feature information corresponding to the mixed semantic image data as output. The classification head may be a linear layer which takes the depth feature information corresponding to the mixed semantic image data as input and the first unknown semantic score, the self-attention map, and the second energy score matrix corresponding to the mixed semantic image data as output. The calibration module may be a convolutional layer composed of two layers of convolutional blocks, which takes the pixel-level feature information corresponding to the mixed semantic image data, the first unknown semantic score, and the self-attention map as input and the calibrated pixel-level feature information corresponding to the mixed semantic image data and the first unknown semantic calibration score as output.

[0044] The third layer may include a semantically relevant decoupler and a semantically irrelevant decoupler. The semantically relevant decoupler may include a feature decoupler and a classification head. The feature decoupler may be a neural network composed of two layers of convolution blocks, which takes the calibrated pixel-level feature information corresponding to the mixed semantic image data as input and the semantically relevant feature information corresponding to the mixed semantic image data as output. For example, the feature decoupler may be a convolutional neural network. The classification head may be a linear layer which takes the calibrated pixel-level feature information corresponding to the mixed semantic image data as input and the second unknown semantic score corresponding to the mixed semantic image data as output. The semantically irrelevant decoupler may be a neural network composed of two layers of convolution blocks, which takes the calibrated pixel-level feature information corresponding to the mixed semantic image data as input and the semantically irrelevant feature information corresponding to the mixed semantic image data as output. For example, the semantically irrelevant decoupler may be a convolutional neural network.

[0045] The fourth layer may be a reconstruction layer. The reconstruction layer may include a splicing function and an image reconstruction network. The splicing function may be a function that takes semantically relevant feature information and semantically irrelevant feature information corresponding to the mixed semantic image data as input, and takes semantic feature information of overlapping areas corresponding to the mixed semantic image data as output. For example, the splicing function may be a tensor splicing function. The image reconstruction network may be a neural network composed of five layers of convolutional blocks and activation functions, taking semantic feature information of overlapping areas corresponding to the mixed semantic image data as input, and taking reconstructed image data corresponding to the mixed semantic image data as output. For example, the image reconstruction network may be a convolutional neural network. The activation function may be a TanH function.

[0046] In step 1032, each depth feature information is input into a micro-decoder in the open world semantic segmentation network to decode each mixed semantic image data to obtain each pixel-level feature information, each first unknown semantic score, each self-attention map and each second energy score matrix.

[0047] In some embodiments, the above-mentioned depth feature information can be input into a micro-decoder in an open-world semantic segmentation network to decode the above-mentioned mixed semantic image data to obtain pixel-level feature information, first unknown semantic scores, self-attention maps, and second energy score matrices. The pixel-level feature information in each of the above-mentioned pixel-level feature information can be a high-dimensional vector composed of feature vectors corresponding to each pixel in the mixed semantic image data. The pixel-level feature information in each of the above-mentioned pixel-level feature information can be deep feature information that more accurately captures local features in the mixed semantic image data after processing by a convolutional layer. The first unknown semantic scores in each of the above-mentioned first unknown semantic scores can be a matrix in which each row corresponds to a row of pixels in the mixed semantic image data, each column corresponds to a column of pixels in the mixed semantic image data, and each element can be a matrix representing the probability value of whether a pixel in the mixed semantic image data is an unknown semantic pixel. The unknown semantic pixels can be pixels whose corresponding semantic labels are unknown. The self-attention maps in each of the above-mentioned self-attention maps can be the self-attention maps generated after the depth feature information is input into the decoding module in the above-mentioned micro-decoder. The self-attention maps in each of the above-mentioned self-attention maps correspond to the depth feature information in each of the above-mentioned depth feature information. The self-attention map can be a matrix in which each row corresponds to a row of pixels in the mixed semantic image data, each column corresponds to a column of pixels in the mixed semantic image data, and each element can be an attention score corresponding to a pixel in the mixed semantic image data. The second energy score matrix in each of the above-mentioned second energy score matrices can be a matrix in which each row corresponds to a row of elements in the depth feature information, each column corresponds to a column of elements in the depth feature information, and each element is a semantic label distribution data corresponding to each element in the depth feature information. The second energy score matrix can be a more accurate first energy score matrix. In practice, the above-mentioned depth feature information can be input into the decoding module in the above-mentioned micro decoder to obtain each pixel-level feature information. The above-mentioned depth feature information can be input into the classification head in the above-mentioned micro decoder to obtain each first unknown semantic score, each self-attention map and each second energy score matrix.

[0048] Step 1033: Generate each calibrated pixel-level feature similarity based on each self-attention map.

[0049] In some embodiments, each calibrated pixel-level feature similarity may be generated based on each of the self-attention maps, wherein the calibrated pixel-level feature similarity in each of the calibrated pixel-level feature similarities may be a calibrated self-attention matrix.

[0050] In some optional implementations of some embodiments, the following steps may be used to generate respective calibrated pixel-level feature similarities based on the aforementioned self-attention maps:

[0051] In the first step, the self-attention maps are averaged to obtain the pixel-level feature similarity to be calibrated. The pixel-level feature similarity to be calibrated can be the self-attention matrix obtained by averaging the self-attention maps. In practice, the sum of the self-attention maps can be determined as the summed self-attention map. Then, the ratio of the summed self-attention map to the number of self-attention maps can be determined as the pixel-level feature similarity to be calibrated.

[0052] In the second step, for each pixel-level feature information in the above pixel-level feature information, perform the following steps:

[0053] The first sub-step is to generate a query matrix, a key matrix and a value matrix based on the above-mentioned pixel-level feature information. In practice, the product of the above-mentioned pixel-level feature information and the preset query weight matrix can be determined as the query matrix. The product of the above-mentioned pixel-level feature information and the preset key weight matrix can be determined as the key matrix. The product of the above-mentioned pixel-level feature information and the preset value weight matrix can be determined as the value matrix. Among them, the above-mentioned query weight matrix can be a network parameter for converting pixel-level feature information into a query matrix during the training of the above-mentioned open-world semantic segmentation network. The above-mentioned key weight matrix can be a network parameter for converting pixel-level feature information into a key matrix during the training of the above-mentioned open-world semantic segmentation network. The above-mentioned value weight matrix can be a network parameter for converting pixel-level feature information into a value matrix during the training of the above-mentioned open-world semantic segmentation network.

[0054] In the second sub-step, the unknown semantic self-attention map is generated based on the query matrix, key matrix, and value matrix using the following formula:

[0055]

[0056] Among them, A n is the above unknown semantic self-attention map. n It can be an attention matrix that captures the semantic relationship between each pixel in the mixed semantic image data. softmax is the softmax function. Q is the query matrix mentioned above. K T is the transpose of the above bond matrix. K is the dimension of the above bond matrix.

[0057] In the third sub-step, the pixel-level feature similarity to be calibrated is calibrated using the following formula to generate a calibrated pixel-level feature similarity:

[0058]

[0059] Among them, A c The pixel-level feature similarity can be calibrated as above. α can be a preset calibration coefficient. The above can be a pre-set value. Here, there is no limitation on the specific setting of the above calibration coefficient. n It can be the above-mentioned unknown semantic self-attention map, and A0 can be the pixel-level feature similarity to be calibrated. The * in can be element-by-element multiplication. As an example: when α is 2, A0 is When α is multiplied by each element in A0, we can get in It can be element-by-element addition. As an example: when the first matrix is The second matrix is When , the sum of the element 1 in the first row and first column of the first matrix and the element 2 in the first row and first column of the second matrix can be determined as the element in the first row and first column of the matrix obtained after element-by-element addition. Similarly, the result obtained after element-by-element addition of the first matrix and the second matrix is

[0060] Step 1034 : Based on the similarities of the respective calibrated pixel-level features, calibrate the respective pixel-level feature information and the respective first unknown semantic scores to obtain the respective calibrated pixel-level feature information and the respective first unknown semantic calibration scores.

[0061] In some embodiments, the pixel-level feature information and the first unknown semantic scores may be calibrated based on the respective calibrated pixel-level feature similarities to obtain calibrated pixel-level feature information and first unknown semantic calibration scores. The calibrated pixel-level feature information in the calibrated pixel-level feature information may be calibrated pixel-level feature information. The first unknown semantic calibration scores in the first unknown semantic calibration scores may be calibrated first unknown semantic scores.

[0062] In some optional implementations of some embodiments, the above-mentioned pixel-level feature information and the above-mentioned first unknown semantic scores may be calibrated based on the above-mentioned respective calibrated pixel-level feature similarities to obtain the respective calibrated pixel-level feature information and the respective first unknown semantic calibration scores through the following steps:

[0063] In the first step, for each of the above pixel-level feature information, weighted processing is performed on the above pixel-level feature information based on the above-mentioned calibrated pixel-level feature similarity using the following formula to obtain calibrated pixel-level feature information:

[0064] fc =A c ×V;

[0065] Among them, f c is the calibrated pixel-level feature information. V is the value matrix corresponding to the pixel-level feature information.

[0066] In the second step, for each of the first unknown semantic scores, perform the following steps:

[0067] The first sub-step is to generate a first unknown semantic score to be calibrated corresponding to the first unknown semantic score based on the first unknown semantic score. The first unknown semantic score to be calibrated may be the first unknown semantic score after maximum pooling and shape change processing. In practice, the first unknown semantic score may be subjected to maximum pooling processing by the maximum pooling technique to obtain a maximum pooling semantic score. The maximum pooling semantic score may be the first unknown semantic score after maximum pooling processing. Then, the maximum pooling semantic score may be subjected to shape change processing by a shape change function to obtain the first unknown semantic score to be calibrated. The shape change function may be a reshape function.

[0068] In the second sub-step, a preliminary calibration detection score is generated based on the calibration pixel-level feature similarity and the first unknown semantic score to be calibrated using the following formula:

[0069] S′=A c ×S m ;

[0070] Wherein, S′ is the preliminary calibration test score. Wherein, the preliminary calibration test score can be the product of the first unknown semantic score to be calibrated and the calibration pixel-level feature similarity corresponding to the first unknown semantic score. S m is the first unknown semantic score to be calibrated.

[0071] In the third step, for each of the generated preliminary calibration test scores, the following steps are performed:

[0072] In the first sub-step, the preliminary calibration test score is normalized using the following formula to obtain a normalized test score:

[0073]

[0074] in, is the normalized detection score above. min is the minimum function. max is the maximum function.

[0075] In the second sub-step, the first unknown semantic score corresponding to the normalized detection score among the first unknown semantic scores is determined as the first unknown semantic score to be processed.

[0076] In the third sub-step, a first unknown semantic calibration score is generated based on the first unknown semantic score to be processed and the normalized detection score using the following formula:

[0077]

[0078] Among them, S c is the calibration score of the first unknown semantics. S is the score of the first unknown semantics to be processed. The * in can be element-wise multiplication. As an example: when When is [3,2,2,1] and S is [1,2,1,2], when multiplying element by element, Multiply the 3 in the first row and first column of S by the 1 in the first row and first column of S, and the resulting 3 is used as S c The first row and first column of the element. And so on, the final S c is [3,4,2,2].

[0079] Step 1035: For each calibrated pixel-level feature information in each calibrated pixel-level feature information, the calibrated pixel-level feature information is input into the semantically relevant decoupler and the semantically irrelevant decoupler in the open-world semantic segmentation network to perform feature decoupling on the pixel-level feature information to obtain semantically relevant feature information, a second unknown semantic score, and semantically irrelevant feature information.

[0080] In some embodiments, for each of the above-mentioned calibrated pixel-level feature information, the calibrated pixel-level feature information can be input into a semantically relevant decoupler and a semantically irrelevant decoupler in an open-world semantic segmentation network to perform feature decoupling on the above-mentioned pixel-level feature information to obtain semantically relevant feature information, a second unknown semantic score, and semantically irrelevant feature information. The above-mentioned semantically relevant feature information can be a feature vector corresponding to a semantically relevant feature in the calibrated pixel-level feature information. The above-mentioned semantically relevant feature can be a feature vector containing key information in the calibrated pixel-level feature information. The above-mentioned key information can be information used to determine the semantic label corresponding to the pixel. The above-mentioned key information can include but is not limited to at least one of the following: shape, texture, and position. The above-mentioned semantically irrelevant feature information can be a feature vector corresponding to a semantically irrelevant feature in the calibrated pixel-level feature information. The above-mentioned semantically irrelevant feature can be a feature vector containing redundant information in the calibrated pixel-level feature information. The above-mentioned redundant information can be information irrelevant to determining the semantic label of the pixel. For example, the above-mentioned redundant information can be brightness. The above-mentioned second unknown semantic score can be that each row can correspond to a row of pixels in the mixed semantic image data, each column can correspond to a column of pixels in the mixed semantic image data, and each element can be a matrix of probability values ​​used to characterize whether a pixel is an unknown semantic pixel. The above-mentioned second unknown semantic score can be a more accurate first unknown semantic score. In practice, the above-mentioned each calibrated pixel-level feature information can be input into the feature decoupler included in the above-mentioned semantic correlation decoupler to obtain the semantic correlation feature information corresponding to each calibrated pixel-level feature information in the above-mentioned each calibrated pixel-level feature information. In practice, the above-mentioned each calibrated pixel-level feature information can be input into the classification head included in the above-mentioned semantic correlation decoupler to obtain the second unknown semantic score corresponding to each calibrated pixel-level feature information in the above-mentioned each calibrated pixel-level feature information. In practice, the above-mentioned each calibrated pixel-level feature information can be input into the above-mentioned semantic irrelevant decoupler to obtain the semantic irrelevant feature information corresponding to each calibrated pixel-level feature information in the above-mentioned each calibrated pixel-level feature information.

[0081] Step 1036 : Generate semantic feature information of each overlapping area based on the obtained semantically relevant feature information and the obtained semantically irrelevant feature information.

[0082] In some embodiments, each overlapping region semantic feature information can be generated based on each obtained semantically relevant feature information and each obtained semantically irrelevant feature information. The overlapping region semantic feature information in the above-mentioned overlapping region semantic feature information can be a feature vector obtained by splicing any semantically relevant feature information with any semantically irrelevant feature information. In practice, the above-mentioned semantically relevant feature information and the above-mentioned semantically irrelevant feature information can be input into the splicing function in the above-mentioned reconstruction layer to obtain the semantic feature information of each overlapping region.

[0083] Step 1037 : Perform image reconstruction processing on the semantic feature information of each overlapping area to obtain each reconstructed image data.

[0084] In some embodiments, image reconstruction processing can be performed on the semantic feature information of each overlapping region to obtain each reconstructed image data. The reconstructed image data in each reconstructed image data may be an image obtained after the semantic feature information of the overlapping region is reconstructed. In practice, the semantic feature information of each overlapping region can be input into the image reconstruction network in the reconstruction layer to perform image reconstruction processing on the semantic feature information of each overlapping region to obtain each reconstructed image data.

[0085] Step 1038: Generate a target loss value based on each first energy score matrix, each second energy score matrix, the above-mentioned first unknown semantic scores, the obtained second unknown semantic scores, each reconstructed image data and semantically irrelevant feature information.

[0086] In some embodiments, a target loss value may be generated based on the first energy score matrices, the second energy score matrices, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data, and the semantically irrelevant feature information. The target loss value may be a loss value of an open-world semantic segmentation network.

[0087] In some optional implementations of some embodiments, a target loss value may be generated based on the first energy score matrices, the second energy score matrices, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data, and the semantically irrelevant feature information through the following steps:

[0088] In the first step, a reconstructed image loss value is generated based on the above-mentioned reconstructed image data, wherein the above-mentioned reconstructed image loss value can be a loss value between the above-mentioned reconstructed image data.

[0089] In practice, for every two mixed semantic image data in the above mixed semantic image dataset, the following steps can be performed:

[0090] First, any one of the two mixed semantic image data can be determined as the first image data, and the other mixed semantic image data can be determined as the second image data.

[0091] Next, the first image data and the second image data may be processed as in steps 1031 to 1037 to obtain four reconstructed image data corresponding to the first image data and the second image data, wherein the four reconstructed image data are respectively expressed as: and

[0092] Among them, z re It can be semantically related feature information. un It can be semantically irrelevant feature information. It may be the semantically relevant feature information corresponding to the first image data. It may be semantically irrelevant feature information corresponding to the first image data. It may be the semantically relevant feature information corresponding to the second image data. It may be semantically irrelevant feature information corresponding to the second image data. Can be due to and The reconstructed image data is composed of Can be due to and The reconstructed image data is composed of Can be due to and The reconstructed image data is composed of Can be due to and The reconstructed image data is composed of

[0093] Then, the loss values ​​of the above four reconstructed image data can be obtained as the reconstruction loss value by the following formula:

[0094]

[0095] Among them, L recon The loss values ​​of the four reconstructed image data can be x 1o It can be the image area in the first image data to which the target image patch is pasted. 2o The second image data may be an image area to which the target image patch is pasted.

[0096] Finally, the sum of the obtained reconstruction loss values ​​can be determined as the above-mentioned reconstructed image loss value.

[0097] In the second step, the semantically irrelevant loss value is generated based on the above semantically irrelevant feature information using the following formula:

[0098]

[0099] Among them, L un is the semantically irrelevant loss value mentioned above. un It can be the loss value between the semantically irrelevant feature information mentioned above. i is the sequence number. E is the symbol used for averaging. is the i-th semantically irrelevant feature information among the above semantically irrelevant feature information. KL Represents relative entropy. N(0,1) represents a normal distribution with mean 0 and variance 1. Used for calculation The relative entropy between N(0,1).

[0100] In the third step, based on each of the first energy score matrices, the following steps are performed:

[0101] In a first sub-step, a second energy score matrix corresponding to the first energy score matrix among the second energy score matrices is determined as a target second energy score matrix.

[0102] In the second sub-step, energy score error data is generated based on the first energy score matrix and the target second energy score matrix using the following formula:

[0103] Ψ(u1,h d ,u2,h o )=|E(u1,h d )-E(u2,h o )| 2 ;

[0104] Among them, Ψ(u1,h d ,u2,h o ) is the energy fraction error data. d ,u2,h o ) may be the square of the absolute value of the difference between the first energy score matrix and the target second energy score matrix. d is the classification header in the above micro decoder. o is the classification head in the semantic segmentation network. u1 is the depth feature information corresponding to the first energy score matrix. u2 is the pixel-level feature information corresponding to the first energy score matrix. E(u1, h d ) is the first energy score matrix. E(u2,h o ) is the above-mentioned target second energy score matrix.

[0105] In a third sub-step, in response to determining that the energy fraction error data satisfies a preset error condition, the energy fraction error data is determined as an energy fraction error value. The preset error condition may be that the energy fraction error data is greater than or equal to a preset error threshold. The error threshold may be a pre-set value. The specific setting of the error threshold is not limited herein.

[0106] In a fourth sub-step, in response to determining that the energy score error data does not satisfy the preset error condition, the preset error threshold is determined as the energy score error value.

[0107] The fifth sub-step is to generate the penalty loss value based on the energy score error value using the following formula:

[0108]

[0109] Among them, L CDL It can be the penalty loss value. H is the number of rows of the first energy score matrix or the target second energy score matrix. W is the number of columns of the first energy score matrix or the target second energy score matrix. i is the index of H. j is the index of W. log is the logarithmic function with base 10. P is the probability value. h p is the classification head in the semantic correlation decoupler. y is the category label. z re is the semantically relevant feature information corresponding to the first energy score matrix and the target second energy score matrix. For classification header h p Output, z re The probability that the category label corresponding to the feature element in the i-th row and j-th column is y is the score value corresponding to the i-th row and j-th column in the second unknown semantic score corresponding to the first energy score matrix above. d is the classification header in the above micro decoder. d is the pixel-level feature information corresponding to the first energy score matrix and the target second energy score matrix. For classification header h d Output, z d The probability that the category label corresponding to the feature element in the i-th row and j-th column is y is the score value corresponding to the i-th row and j-th column in the first unknown semantic score corresponding to the first energy score matrix. d ,u2,h o ) i，j is the element value corresponding to the i-th row and j-th column in the above energy score error data.

[0110] In the fourth step, the sum of the above-mentioned reconstructed image loss value, the above-mentioned semantically irrelevant loss value and the generated penalty item loss values ​​is determined as the target loss value.

[0111] Step 1039, in response to determining that the target loss value satisfies the preset loss value condition, determining the open-world semantic segmentation network as the open-world semantic segmentation model, and determining the respective first unknown semantic calibration scores obtained in the last training step as the respective target calibration scores.

[0112] In some embodiments, in response to determining that the target loss value satisfies a preset loss value condition, the open-world semantic segmentation network can be determined as an open-world semantic segmentation model, and each first unknown semantic calibration score obtained in the last training step can be determined as each target calibration score. The preset loss value condition can be that the target loss value is less than a preset loss value threshold. The loss value threshold can be a pre-set value. The specific setting of the loss value threshold is not limited here.

[0113] Step 10310, in response to determining that the target loss value does not meet the preset loss value condition, adjust the network parameters of the open-world semantic segmentation network, and use unused mixed semantic image data to form a mixed semantic image dataset, use the adjusted open-world semantic segmentation network as the open-world semantic segmentation network, and perform the training step again.

[0114] In some embodiments, in response to determining that the target loss value does not satisfy the preset loss value condition, network parameters of the open-world semantic segmentation network can be adjusted, and unused mixed semantic image data can be used to form a mixed semantic image dataset. The adjusted open-world semantic segmentation network can then be used as the open-world semantic segmentation network to perform the training step again. In practice, a back propagation algorithm (BP algorithm) and a gradient descent method (e.g., a mini-batch gradient descent algorithm) can be used to adjust the network parameters of the open-world semantic segmentation network.

[0115] Figure 2 A schematic diagram of generating various mixed semantic image data based on various known semantic image data may be provided for the open-world semantic segmentation model construction method according to some embodiments of the present disclosure.

[0116] The known semantic image data may be the known semantic image data described above. Random scaling may be a process of randomly scaling any one of the known semantic image data. Color enhancement may be a process of color enhancing the randomly scaled known semantic image data to obtain a preprocessed sample image. Image cropping may be a process of cropping the preprocessed sample image based on the shape information to obtain image patches. The image patches may be the image patches described above. The sample image may be the sample image described above. Random scaling may be a process of randomly scaling the sample image. Random pasting may be a process of pasting the image patches onto the sample image to obtain patch sample images. The patch sample images may be the patch sample images described above. "Cropping with overlapping areas" may be a process of cropping the patch sample images based on the cropping information to obtain cropped sample images. The cropped sample images may be the cropped sample images described above. Random masking may be a process of randomly masking the cropped sample images to obtain masked sample images. The masked sample images may be the masked sample images described above. Pasting in overlapping areas can be a process of pasting the target image patches to preset locations in the mask sample images to obtain mixed semantic image data. Color dithering can be a process of pasting target image patches that meet preset pasting conditions to preset locations in the mask sample images, and then performing brightness adjustment, saturation adjustment, hue adjustment, and other processing on the mask sample images after the target image patches are pasted using data enhancement techniques.

[0117] Figure 3 A schematic diagram of generating calibrated pixel-level feature similarity, calibrated pixel-level feature information, and a first unknown semantic calibration score for an open-world semantic segmentation model construction method according to some embodiments of the present disclosure.

[0118] Wherein, S can represent the first unknown semantic score and the first unknown semantic score to be processed. The maximum pooling and shape change can be performed on S to obtain the first unknown semantic score to be calibrated S. m process. m It can represent the first unknown semantic score to be calibrated. S′ can represent the preliminary calibration detection score. Normalization can be performed on each S′ to obtain a normalized detection score. process. c Can be for The first unknown semantic calibration score obtained by element-wise multiplication of and S. The self-attention map A from the frozen pre-training branch i, where the frozen pre-trained branch is the micro decoder mentioned above, A i The self-attention map output by the micro decoder can be the average of A i The process of performing average processing to obtain the pixel-level feature similarity A0 to be calibrated. α can be the above-mentioned calibration coefficient. The pixel-level features of the new branch can be the pixel-level feature information output by the above-mentioned micro decoder. Shape transformation projection can be the process of performing shape transformation, projection and other processing on the pixel-level feature information to obtain the query matrix Q, key matrix K and value matrix V. Among them, the query matrix Q and key matrix K can be used to generate the unknown semantic self-attention map A n Then through A n , A0 and α to obtain the calibrated pixel-level feature similarity A c .f c Can be passed A c The calibrated pixel-level feature information obtained by and V. Matrix multiplication It corresponds to × in the formula. Add element by element It can correspond to the formula Element-wise multiplication It can correspond to * in the formula.

[0119] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for constructing an open-world semantic segmentation model, comprising: Based on the preset known semantic image data, generating various mixed semantic image data; Determining a preset number of mixed semantic image data among the mixed semantic image data as a mixed semantic image dataset; Based on the mixed semantic image dataset, the following training steps are performed: Inputting the mixed semantic image dataset into the semantic segmentation network in the open world semantic segmentation network to obtain each depth feature information and each first energy score matrix; Inputting the respective deep feature information into a micro-decoder in an open-world semantic segmentation network to decode the respective mixed semantic image data to obtain respective pixel-level feature information, respective first unknown semantic scores, respective self-attention maps, and respective second energy score matrices; Generating respective calibrated pixel-level feature similarities based on the respective self-attention maps; Based on the respective calibrated pixel-level feature similarities, calibrating the respective pixel-level feature information and the respective first unknown semantic scores to obtain respective calibrated pixel-level feature information and respective first unknown semantic calibration scores; For each calibrated pixel-level feature information, input the calibrated pixel-level feature information into a semantically relevant decoupler and a semantically irrelevant decoupler in an open-world semantic segmentation network to perform feature decoupling on the calibrated pixel-level feature information to obtain semantically relevant feature information, a second unknown semantic score, and semantically irrelevant feature information. Generate semantic feature information of each overlapping area based on the obtained semantically relevant feature information and the obtained semantically irrelevant feature information; Performing image reconstruction processing on the semantic feature information of each overlapping area to obtain each reconstructed image data; generating a target loss value based on the first energy score matrices, the second energy score matrices, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data, and the semantically irrelevant feature information; In response to determining that the target loss value satisfies a preset loss value condition, determining the open-world semantic segmentation network as an open-world semantic segmentation model, and determining each first unknown semantic calibration score obtained in the last training step as each target calibration score; In response to determining that the target loss value does not satisfy the preset loss value condition, the network parameters of the open-world semantic segmentation network are adjusted, and unused mixed semantic image data are used to form a mixed semantic image dataset, and the adjusted open-world semantic segmentation network is used as the open-world semantic segmentation network to perform the training step again.

2. The method according to claim 1, wherein The generating of each mixed semantic image data based on each preset known semantic image data includes: Randomly selecting a known semantic image data from each known semantic image data acquired in advance as a known semantic sample image; Preprocessing the known semantic sample image to obtain a preprocessed sample image; Based on the preset shape information, the preprocessed sample image is cropped to obtain image patches; Randomly pasting each of the image patches onto a preset sample image to obtain a patch sample image; Based on preset cropping information, cropping the patch sample image to obtain individual cropped sample images; Performing random masking on each of the cropped sample images to obtain each masked sample image; determining image patches that meet a preset shape condition among the image patches as target image patches; For each mask sample image in the mask sample images, a target image patch that meets a preset pasting condition in the target image patches is pasted to a preset position in the mask sample image to obtain mixed semantic image data.

3. The method according to claim 1, wherein Generating each calibrated pixel-level feature similarity based on each self-attention map includes: Averaging the self-attention maps to obtain the pixel-level feature similarity to be calibrated; For each piece of pixel-level feature information, perform the following steps: Based on the pixel-level feature information, generating a query matrix, a key matrix and a value matrix; The unknown semantic self-attention map is generated based on the query matrix, key matrix and value matrix by the following formula: Among them, A n is the unknown semantic self-attention map, softmax is the softmax function, Q is the query matrix, K T is the transpose of the bond matrix, d K is the dimension of the bond matrix; The pixel-level feature similarity to be calibrated is calibrated using the following formula to generate a calibrated pixel-level feature similarity: Among them, A c is the calibration pixel-level feature similarity, α is the preset calibration coefficient, A n is the unknown semantic self-attention map, A0 is the pixel-level feature similarity to be calibrated, The * in is element-by-element multiplication. in It is element-wise addition.

4. The method according to claim 3, wherein: The step of calibrating the respective pixel-level feature information and the respective first unknown semantic scores based on the respective calibrated pixel-level feature similarities to obtain the respective calibrated pixel-level feature information and the respective first unknown semantic calibration scores includes: For each pixel-level feature information in the respective pixel-level feature information, weighted processing is performed on the pixel-level feature information based on the calibrated pixel-level feature similarity using the following formula to obtain calibrated pixel-level feature information: f c =A c ×V; Among them, f c is the calibration pixel-level feature information, and V is the value matrix corresponding to the pixel-level feature information; For each of the first unknown semantic scores, performing the following steps: generating, based on the first unknown semantic score, a first unknown semantic score to be calibrated corresponding to the first unknown semantic score; A preliminary calibration detection score is generated based on the calibration pixel-level feature similarity and the first unknown semantic score to be calibrated using the following formula: S′=A c ×S m ; Where S′ is the preliminary calibration test score, S m is the first unknown semantic score to be calibrated; For each of the generated preliminary calibration test scores, perform the following steps: The preliminary calibration test score is normalized by the following formula to obtain a normalized test score: in, is the normalized detection score, min is the minimum function, and max is the maximum function; Determining a first unknown semantic score corresponding to the normalized detection score among the first unknown semantic scores as a first unknown semantic score to be processed; A first unknown semantic calibration score is generated based on the first unknown semantic score to be processed and the normalized detection score using the following formula: Among them, S c is the first unknown semantic calibration score, S is the first unknown semantic score to be processed, The * in the string is element-wise multiplication.

5. The method according to claim 1, wherein The generating a target loss value based on the respective first energy score matrices, the respective second energy score matrices, the respective first unknown semantic scores, the obtained respective second unknown semantic scores, the respective reconstructed image data, and the semantically irrelevant feature information includes: generating a reconstructed image loss value based on the respective reconstructed image data; The semantically irrelevant loss value is generated based on the semantically irrelevant feature information using the following formula: Among them, L un is the semantically irrelevant loss value, u is the serial number, E is the symbol used for averaging, is the i-th semantically irrelevant feature information among the semantically irrelevant feature information, D KL represents relative entropy, N(0,1) represents a normal distribution with a mean of 0 and a variance of 1, Used for calculation Relative entropy with N(0,1); Based on each of the first energy score matrices, the following steps are performed: determining a second energy score matrix corresponding to the first energy score matrix among the second energy score matrices as a target second energy score matrix; Energy score error data is generated based on the first energy score matrix and the target second energy score matrix using the following formula: Ψ(u1,h d ,u2,h o )=|E(u1,h d )-E(u2,h o )| 2 ; Among them, Ψ(u1,h d ,u2,h o ) is the energy fraction error data, h d is the classification head in the micro decoder, h o is the classification head in the semantic segmentation network, u1 is the depth feature information corresponding to the first energy score matrix, u2 is the pixel-level feature information corresponding to the first energy score matrix, E(u1, h d ) is the first energy score matrix, E(u2,h o ) is the target second energy score matrix; In response to determining that the energy fraction error data satisfies a preset error condition, determining the energy fraction error data as an energy fraction error value; In response to determining that the energy score error data does not satisfy the preset error condition, determining a preset error threshold as an energy score error value; The penalty loss value is generated based on the energy score error value using the following formula: Wherein, H is the number of rows of the first energy score matrix or the target second energy score matrix, W is the number of columns of the first energy score matrix or the target second energy score matrix, i is the index of H, j is the index of W, log is the logarithmic function with base 10, P is the probability value, h p is the classification head in the semantic correlation decoupler, y is the category label, z re is the semantically relevant feature information corresponding to the first energy score matrix and the target second energy score matrix, For classification header h p Output, z re The probability that the category label corresponding to the feature element in the i-th row and j-th column is y is the score value corresponding to the i-th row and j-th column in the second unknown semantic score corresponding to the first energy score matrix, h d is the classification head in the micro decoder, z d is the pixel-level feature information corresponding to the first energy score matrix and the target second energy score matrix, For classification header h d Output, z d The probability that the category label corresponding to the feature element in the i-th row and j-th column is y is the score value corresponding to the i-th row and j-th column in the first unknown semantic score corresponding to the first energy score matrix, Ψ(u1,h d ,u2,h o ) i，j is the element value corresponding to the i-th row and j-th column in the energy score error data; The sum of the reconstructed image loss value, the semantically irrelevant loss value, and the generated penalty item loss values ​​is determined as a target loss value.

Images