Method for constructing semantic segmentation model of open world

Abstract

The embodiment of the invention discloses an open world semantic segmentation model construction method. A specific embodiment of the method comprises the following steps: generating each piece of mixed semantic image data; determining a mixed semantic image data set; obtaining each piece of depth feature information and each first energy score matrix; obtaining each piece of pixel-level feature information, each first unknown semantic score, each self-attention map and each second energy score matrix; generating the similarity of each calibration pixel level feature; obtaining each piece of calibration pixel-level feature information and each first unknown semantic calibration score; obtaining semantic related feature information, a second unknown semantic score and semantic independent feature information; generating semantic feature information of each overlapping region; obtaining each piece of reconstructed image data; generating a target loss value; and determining the open world semantic segmentation network as an open world semantic segmentation model. According to the embodiment, a more reliable open world semantic segmentation model can be trained, and the recognition accuracy of the model can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method for constructing an open world semantic segmentation model. Background Art

[0002] For many applications such as autonomous driving, medical image analysis, and intelligent monitoring, accurate semantic segmentation has a crucial impact on the effect and performance of the application. Only by accurately segmenting different semantic regions in the image can obstacles be accurately identified and avoided in autonomous driving, and lesion areas can be accurately identified and labeled in medical images. However, traditional image processing methods often rely on manually designed features and rules, which are difficult to adapt to complex and changing scenes; semantic segmentation methods based on deep learning can automatically extract rich feature information by learning a large amount of data, and have higher robustness and adaptability. With the development of deep learning, especially convolutional neural networks, semantic segmentation technology has made significant progress in accuracy and efficiency. Compared with other methods, semantic segmentation based on deep learning not only has a significant improvement in accuracy, but also has the advantages of end-to-end training, high degree of automation, and strong adaptability. Therefore, it has a wide range of application scenarios and great research value in the fields of autonomous driving, medical imaging, and intelligent monitoring.

[0003] Existing semantic segmentation technologies usually train models on closed-set data (within the distribution). These models can only recognize categories defined within the distribution and cannot recognize categories that have not been seen during training, that is, unknown semantic (outside the distribution) inputs. This causes the model to identify unknown categories as known categories when faced with unknown inputs, posing a huge safety hazard. For example, a semantic segmentation model trained on a closed-set street view dataset cannot recognize an unknown object when faced with an unknown obstacle in an open scene, posing a serious threat to the safety of self-driving cars. Therefore, it is crucial to accurately identify pixels of unknown semantic categories while accurately segmenting pixels of categories within the distribution. This can be defined as: open-world semantic segmentation, that is, on the basis of known categories, an effective mechanism or method is used to detect and distinguish unknown semantic categories to improve the safety and reliability of the model in practical applications.

[0004] Pioneers have conducted research on open-world semantic segmentation technology to varying degrees. W. Liu et al. measured whether each pixel was an unknown semantic pixel by estimating the uncertainty of the neural network output, and used energy scores, Euclidean distances, entropy and other metrics to evaluate uncertainty. This method does not require retraining the neural network, but it cannot effectively distinguish difficult samples that are similar to those in the distribution. Y. Xia et al. reconstructed the features or semantic graphs output by the neural network, compared the reconstruction results with the original image, and regarded the areas with large reconstruction errors as unknown semantic pixels. Since the reconstruction network will overfit the appearance of the data in the distribution, this method is difficult to generalize to scenes with large domain offsets, and the reconstruction network introduces a large amount of additional computation. Y. Liu et al. introduced additional data sets as unknown data, cut and pasted the additional data sets onto the data sets in the distribution, and added modules to the closed set pre-trained model to fit the unknown data, so that the model has the ability to identify open set detection. Although this method has good unknown semantic recognition capabilities, it introduces a large number of additional annotated data sets, which usually require huge manpower and material resources to obtain, and are even difficult to obtain.

[0005] Technical issues: Existing semantic segmentation technologies usually train models on known data (within the distribution). These models can only recognize categories defined by known data, and cannot recognize data that has not been seen during training, that is, unknown semantic data (out of the distribution) input. This causes the model to recognize unknown semantic data as known data when faced with input of unknown semantic data, resulting in low reliability and accuracy of model recognition. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose an open-world semantic segmentation model construction method to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a method for constructing an open-world semantic segmentation model, the method comprising: generating each mixed semantic image data based on each preset known semantic image data; determining a preset number of mixed semantic image data in the above-mentioned each mixed semantic image data as a mixed semantic image data set; performing the following training steps based on the mixed semantic image data set: inputting the mixed semantic image data set into a semantic segmentation network in an open-world semantic segmentation network to obtain each deep feature information and each first energy score matrix; inputting the above-mentioned each deep feature information into a micro decoder in the open-world semantic segmentation network to decode the above-mentioned each mixed semantic image data to obtain each pixel-level feature information, each first unknown semantic score, each self-attention map and each second energy score matrix; generating each calibrated pixel-level feature similarity based on the above-mentioned each self-attention map; calibrating the above-mentioned each pixel-level feature information and the above-mentioned each first unknown semantic score based on the above-mentioned each calibrated pixel-level feature similarity to obtain each calibrated pixel-level feature information and each first unknown semantic calibration score; for each calibrated pixel-level feature information in the above-mentioned each calibrated pixel-level feature information, inputting the above-mentioned calibrated pixel-level feature information into the open-world semantic segmentation network. The semantically relevant decoupler and the semantically irrelevant decoupler in the method are used to perform feature decoupling on the calibrated pixel-level feature information to obtain semantically relevant feature information, second unknown semantic scores and semantically irrelevant feature information; based on the obtained semantically relevant feature information and the obtained semantically irrelevant feature information, each overlapping area semantic feature information is generated; image reconstruction processing is performed on the semantic feature information of each overlapping area to obtain each reconstructed image data; based on the first energy score matrix, the second energy score matrix, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data and the semantically irrelevant feature information, a target loss value is generated; in response to determining that the target loss value satisfies a preset loss value condition, the open-world semantic segmentation network is determined as an open-world semantic segmentation model, and the first unknown semantic calibration scores obtained in the last training step are determined as target calibration scores; in response to determining that the target loss value does not satisfy the preset loss value condition, the network parameters of the open-world semantic segmentation network are adjusted, and the unused mixed semantic image data are used to form a mixed semantic image data set, and the adjusted open-world semantic segmentation network is used as the open-world semantic segmentation network to perform the training step again.

[0009] The above-mentioned embodiments of the present disclosure have the following beneficial effects:

[0010] 1. The present invention discloses an open-world semantic segmentation algorithm. Without introducing additional data, a large amount of unknown semantic data is synthesized using in-distribution data, thereby alleviating the problem of missing unknown semantic data. Through the self-supervised training method of "decoupling + reconstruction", the overfitting problem of the model on the generated data is alleviated, greatly improving the ability of the semantic segmentation model to recognize unknown inputs.

[0011] 2. The present invention proposes a dynamic penalty loss function, which dynamically penalizes the model by estimating the possibility that the pixel is of unknown semantics, thereby alleviating the label noise introduced by synthetic data; at the same time, the present invention proposes a pixel-to-pixel feature similarity calibration module, which calibrates the inaccurate pixel-to-pixel feature similarity by hybrid self-attention, clarifies the decision boundary between distributions, and improves the model's ability to identify unknown semantic pixels.

[0012] 3. The method proposed in the present invention can be migrated to deep learning semantic segmentation models of various architectures, and has the characteristics of simple design, high robustness, and high detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the method for building an open world semantic segmentation model according to the present disclosure;

[0015] Figure 2 A schematic diagram of generating various mixed semantic image data based on various known semantic image data according to an open world semantic segmentation model construction method of some embodiments of the present disclosure;

[0016] Figure 3 A schematic diagram of generating calibrated pixel-level feature similarity, calibrated pixel-level feature information and a first unknown semantic calibration score according to an open-world semantic segmentation model construction method according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 A process 100 of some embodiments of the method for building an open world semantic segmentation model according to the present disclosure is shown. The method for building an open world semantic segmentation model comprises the following steps:

[0024] Step 101: Generate various mixed semantic image data based on various preset known semantic image data.

[0025] In some embodiments, various mixed semantic image data can be generated based on various preset known semantic image data. Among them, the known semantic image data in the above-mentioned various known semantic image data can be image data in which each pixel point is a known semantic pixel point. The above-mentioned known semantic pixel point can be a pixel point whose corresponding semantic label is known. The above-mentioned semantic label can be a label for characterizing the category corresponding to the pixel point. The mixed semantic image data in the above-mentioned various mixed semantic image data can be image data containing known semantic pixel points and unknown semantic pixel points. The above-mentioned unknown semantic pixel point can be a pixel point whose corresponding semantic label is unknown.

[0026] In some optional implementations of some embodiments, various mixed semantic image data may be generated based on various preset known semantic image data through the following steps:

[0027] In the first step, a known semantic image data is randomly selected from each known semantic image data acquired in advance as a known semantic sample image.

[0028] The second step is to preprocess the known semantic sample image to obtain a preprocessed sample image. In practice, the known semantic sample image can be randomly scaled, and the randomly scaled image can be color enhanced by an image enhancement technique to obtain a preprocessed sample image. The image enhancement technique can be false color processing.

[0029] The third step is to crop the preprocessed sample image based on the preset shape information to obtain image patches. The shape information in the shape information can be used to characterize the shape and size of the cropped image when the preprocessed sample image is cropped. For example, the shape information may be a square image with a size of 20*20. The image patch in the image patches may be an image area in the preprocessed sample image. In practice, for each shape information in the shape information, the preprocessed sample image may be randomly cropped into the shape and size corresponding to the shape information through the image processing library to obtain an image patch. The image processing library may be the OpenCV library.

[0030] The fourth step is to randomly paste the above-mentioned image patches to a preset sample image to obtain a patch sample image. The above-mentioned sample image can be any known semantic image data other than the above-mentioned known semantic sample image among the above-mentioned known semantic image data. The above-mentioned patch sample image can be a sample image to which each image patch is pasted. In practice, the above-mentioned image patches can be randomly pasted to the above-mentioned sample image through an image processing library to obtain a patch sample image. The above-mentioned image processing library can be an OpenCV library.

[0031] Optionally, before randomly pasting the above-mentioned image patches to a preset sample image to obtain a patch sample image, the above-mentioned sample image may be randomly scaled to obtain a randomly scaled sample image as a sample image to update the sample image. Then, the above-mentioned image patches may be randomly pasted to the updated sample image to obtain a patch sample image.

[0032] The fifth step is to crop the patch sample image based on the preset cropping information to obtain each cropped sample image. The cropping information may be information used to characterize the method used to crop the patch sample image. For example, the cropping information may be "cropping using a preset number of rectangles with overlapping parts". The cropped sample images in each of the cropped sample images may be images obtained by cropping the patch sample image. Every two cropped sample images in each of the cropped sample images have the same image area. In practice, the patch sample image may be cropped according to the method corresponding to the cropping information through a library function to obtain each cropped sample image corresponding to the patch sample image. The library function may be crop().

[0033] The sixth step is to perform random masking on each of the above cropped sample images to obtain each masked sample image. Among them, the masked sample image in each of the above masked sample images can be an image obtained after randomly masking the cropped sample image. Every two masked sample images in each of the above masked sample images have the same image area. In practice, for each of the above cropped sample images, the pixel values ​​corresponding to the pixel points of a preset number of pixels in the above cropped sample image can be set to preset pixel values ​​through the above image processing library to obtain a masked sample image. The above preset number of pixels can be a preset value. Here, there is no limitation on the specific setting of the above preset number of pixels. The above preset pixel value can be a value between 0 and 255. Here, there is no limitation on the specific setting of the above preset pixel value.

[0034] In the seventh step, the image patches that meet the preset shape condition among the above image patches are determined as target image patches. The preset shape condition may be that the shapes of the image patches are the same.

[0035] In the eighth step, for each of the mask sample images above, the target image patches among the target image patches that meet the preset pasting conditions are pasted to the preset positions in the mask sample images above to obtain mixed semantic image data. The preset positions may be any image regions among the image regions in the mask sample images that are the same as the target mask sample images. The target mask sample images may be mask sample images other than the mask sample images among the mask sample images above. In practice, for each of the mask sample images above, the target image patches among the target image patches that meet the preset pasting conditions are pasted to the preset positions in the mask sample images above through the image processing library to obtain mixed semantic image data.

[0036] Optionally, after pasting the target image patches satisfying the preset pasting conditions in the above-mentioned target image patches to the preset positions in the above-mentioned mask sample images, the mask sample images after pasting the target image patches can also be subjected to brightness adjustment, saturation adjustment, hue adjustment and other processing by data enhancement technology to obtain respective adjusted mask sample images. Among them, the adjusted mask sample images in the above-mentioned adjusted mask sample images can be mask sample images subjected to brightness adjustment, saturation adjustment, hue adjustment and other processing. Then, the above-mentioned adjusted mask sample images can be determined as respective mixed semantic image data.

[0037] Step 102: determine a preset number of mixed semantic image data in each mixed semantic image data as a mixed semantic image data set.

[0038] In some embodiments, a preset number of mixed semantic image data in each of the mixed semantic image data may be determined as a mixed semantic image data set. The preset number of images may be a preset value. Here, the specific setting of the preset number of images is not limited.

[0039] Step 103, based on the mixed semantic image dataset, perform the following training steps:

[0040] Step 1031 , input the mixed semantic image dataset into a semantic segmentation network in an open world semantic segmentation network to obtain various deep feature information and various first energy score matrices.

[0041] In some embodiments, the mixed semantic image data set can be input into the semantic segmentation network in the open world semantic segmentation network to obtain various depth feature information and various first energy score matrices. The depth feature information in the above-mentioned various depth feature information can be a feature vector corresponding to the mixed semantic image data. The first energy score matrix in the above-mentioned various first energy score matrices can be a matrix of semantic label distribution data corresponding to the pixel points in the mixed semantic image data, each row can correspond to a row of pixels in the mixed semantic image data, each column can correspond to a column of pixels in the mixed semantic image data, and each element can be a matrix of semantic label distribution data corresponding to the pixel points in the mixed semantic image data. The above-mentioned semantic label distribution data can be data used to characterize the probability that the pixel points belong to each semantic label. For example, the above-mentioned semantic label distribution data can be "semantic label: sky, probability: 90%; semantic label: earth, probability: 10%". In practice, the above-mentioned mixed semantic image data set can be input into the encoder in the semantic segmentation network to obtain the multi-scale features corresponding to each mixed semantic image data in the above-mentioned mixed semantic image data set. Then, the obtained multi-scale features can be input into the decoder in the above-mentioned semantic segmentation network to obtain the depth feature information corresponding to each mixed semantic image data in the above-mentioned mixed semantic image data set. Finally, the obtained deep feature information can be input into the classification head in the semantic segmentation network to obtain the first energy score matrix corresponding to each mixed semantic image data in the mixed semantic image data set. The open world semantic segmentation network can include four layers.

[0042] The first layer may be a semantic segmentation network. The semantic segmentation network may include an encoder, a decoder and a classification head. The encoder may be a neural network that takes the mixed semantic image data as input and outputs the multi-scale features corresponding to the mixed semantic image data. For example, the encoder may be a convolutional neural network. The decoder may be a neural network that takes the multi-scale features corresponding to the mixed semantic image data as input and outputs the depth feature information corresponding to the mixed semantic image data. For example, the decoder may be a convolutional neural network. The classification head may be a linear layer that takes the depth feature information corresponding to the mixed semantic image data as input and outputs the first energy score matrix corresponding to the mixed semantic image data.

[0043] The second layer may be a micro-decoder. The micro-decoder may include a decoding module, a classification head and a calibration module. The decoding module may be a convolutional layer composed of three layers of convolutional blocks, which takes the deep feature information corresponding to the mixed semantic image data as input and the pixel-level feature information corresponding to the mixed semantic image data as output. The classification head may be a linear layer which takes the deep feature information corresponding to the mixed semantic image data as input and the first unknown semantic score, the self-attention map and the second energy score matrix corresponding to the mixed semantic image data as output. The calibration module may be a convolutional layer composed of two layers of convolutional blocks, which takes the pixel-level feature information corresponding to the mixed semantic image data, the first unknown semantic score and the self-attention map as input and the calibrated pixel-level feature information corresponding to the mixed semantic image data and the first unknown semantic calibration score as output.

[0044] The third layer may include a semantically relevant decoupler and a semantically irrelevant decoupler. The semantically relevant decoupler may include a feature decoupler and a classification head. The feature decoupler may be a neural network composed of two layers of convolution blocks, which takes the calibrated pixel-level feature information corresponding to the mixed semantic image data as input and the semantically relevant feature information corresponding to the mixed semantic image data as output. For example, the feature decoupler may be a convolutional neural network. The classification head may be a linear layer which takes the calibrated pixel-level feature information corresponding to the mixed semantic image data as input and the second unknown semantic score corresponding to the mixed semantic image data as output. The semantically irrelevant decoupler may be a neural network composed of two layers of convolution blocks, which takes the calibrated pixel-level feature information corresponding to the mixed semantic image data as input and the semantically irrelevant feature information corresponding to the mixed semantic image data as output. For example, the semantically irrelevant decoupler may be a convolutional neural network.

[0045] The fourth layer may be a reconstruction layer. The reconstruction layer may include a splicing function and an image reconstruction network. The splicing function may be a function that takes the semantically relevant feature information and the semantically irrelevant feature information corresponding to the mixed semantic image data as input, and takes the semantic feature information of the overlapping area corresponding to the mixed semantic image data as output. For example, the splicing function may be a tensor splicing function. The image reconstruction network may be a neural network composed of five layers of convolutional blocks and activation functions, taking the semantic feature information of the overlapping area corresponding to the mixed semantic image data as input, and taking the reconstructed image data corresponding to the mixed semantic image data as output. For example, the image reconstruction network may be a convolutional neural network. The activation function may be a TanH function.

[0046] Step 1032, input each deep feature information into the micro decoder in the open world semantic segmentation network to decode each mixed semantic image data to obtain each pixel-level feature information, each first unknown semantic score, each self-attention map and each second energy score matrix.

[0047] In some embodiments, the above-mentioned various deep feature information can be input into a micro decoder in an open world semantic segmentation network to decode the above-mentioned various mixed semantic image data to obtain various pixel-level feature information, various first unknown semantic scores, various self-attention maps, and various second energy score matrices. Among them, the pixel-level feature information in the above-mentioned various pixel-level feature information can be a high-dimensional vector composed of various feature vectors corresponding to each pixel in the mixed semantic image data. The pixel-level feature information in the above-mentioned various pixel-level feature information can be deep feature information that captures local features in the mixed semantic image data more accurately after being processed by a convolution layer. The first unknown semantic score in the above-mentioned various first unknown semantic scores can be a row of pixels in the mixed semantic image data corresponding to each row, a column of pixels in the mixed semantic image data corresponding to each column, and each element can be a matrix of probability values ​​for characterizing whether a pixel in the mixed semantic image data is an unknown semantic pixel. The above-mentioned unknown semantic pixel can be a pixel whose corresponding semantic label is unknown. The self-attention map in the above-mentioned various self-attention maps can be a self-attention map generated after the deep feature information is input into the decoding module in the above-mentioned micro decoder. The self-attention maps in each of the above self-attention maps correspond to the depth feature information in each of the above depth feature information. The self-attention map can be a matrix in which each row corresponds to a row of pixels in the mixed semantic image data, each column corresponds to a column of pixels in the mixed semantic image data, and each element can be an attention score corresponding to a pixel in the mixed semantic image data. The second energy score matrix in each of the above second energy score matrices can be a matrix in which each row corresponds to a row of elements in the depth feature information, each column corresponds to a column of elements in the depth feature information, and each element is a semantic label distribution data corresponding to each element in the depth feature information. The second energy score matrix can be a more accurate first energy score matrix. In practice, the above-mentioned each of the depth feature information can be input into the decoding module in the above-mentioned micro decoder to obtain each pixel-level feature information. The above-mentioned each of the depth feature information can be input into the classification head in the above-mentioned micro decoder to obtain each first unknown semantic score, each self-attention map and each second energy score matrix.

[0048] Step 1033, based on each self-attention map, generate each calibrated pixel-level feature similarity.

[0049] In some embodiments, each calibrated pixel-level feature similarity may be generated based on each of the self-attention maps, wherein the calibrated pixel-level feature similarity in each of the calibrated pixel-level feature similarities may be a calibrated self-attention matrix.

[0050] In some optional implementations of some embodiments, each calibration pixel-level feature similarity may be generated based on each of the self-attention maps through the following steps:

[0051] In the first step, the above-mentioned self-attention maps are averaged to obtain the pixel-level feature similarity to be calibrated. Among them, the above-mentioned pixel-level feature similarity to be calibrated can be the self-attention matrix obtained by averaging the above-mentioned self-attention maps. In practice, first, the sum of the above-mentioned self-attention maps can be determined as the summed self-attention map. Then, the ratio of the above-mentioned summed self-attention map to the number of each self-attention map can be determined as the pixel-level feature similarity to be calibrated.

[0052] In the second step, for each pixel-level feature information in the above-mentioned pixel-level feature information, the following steps are performed:

[0053] The first sub-step is to generate a query matrix, a key matrix and a value matrix based on the above-mentioned pixel-level feature information. In practice, the product of the above-mentioned pixel-level feature information and the preset query weight matrix can be determined as the query matrix. The product of the above-mentioned pixel-level feature information and the preset key weight matrix can be determined as the key matrix. The product of the above-mentioned pixel-level feature information and the preset value weight matrix can be determined as the value matrix. Among them, the above-mentioned query weight matrix can be a network parameter used to convert the pixel-level feature information into a query matrix during the training of the above-mentioned open-world semantic segmentation network. The above-mentioned key weight matrix can be a network parameter used to convert the pixel-level feature information into a key matrix during the training of the above-mentioned open-world semantic segmentation network. The above-mentioned value weight matrix can be a network parameter used to convert the pixel-level feature information into a value matrix during the training of the above-mentioned open-world semantic segmentation network.

[0054] In the second sub-step, the unknown semantic self-attention map is generated based on the above query matrix, key matrix and value matrix through the following formula:

[0055]

[0056] Among them, A n is the above unknown semantic self-attention map. n It can be an attention matrix that captures the semantic relationship between each pixel in the mixed semantic image data. softmax is the softmax function. Q is the query matrix above. K T is the transpose of the above key matrix. K is the dimension of the key matrix above.

[0057] In the third sub-step, the pixel-level feature similarity to be calibrated is calibrated by the following formula to generate the calibrated pixel-level feature similarity:

[0058]

[0059] Among them, A c The pixel-level feature similarity can be calibrated as above. α can be a preset calibration coefficient. The above can be a preset value. Here, the specific setting of the above calibration coefficient is not limited. n It can be the above-mentioned unknown semantic self-attention map, and A0 can be the pixel-level feature similarity to be calibrated. The * in can be element-by-element multiplication. As an example: when α is 2, A0 is When α is multiplied by each element in A0, we get In It can be element-by-element addition. As an example: when the first matrix is The second matrix is , the sum of the element 1 in the first row and first column of the first matrix and the element 2 in the first row and first column of the second matrix can be determined as the element in the first row and first column of the matrix obtained by element-by-element addition. Similarly, the result obtained by element-by-element addition of the first matrix and the second matrix is

[0060] Step 1034: calibrate each pixel-level feature information and each first unknown semantic score based on each calibration pixel-level feature similarity to obtain each calibration pixel-level feature information and each first unknown semantic calibration score.

[0061] In some embodiments, the above-mentioned pixel-level feature information and the above-mentioned first unknown semantic scores can be calibrated based on the above-mentioned calibrated pixel-level feature similarities to obtain each calibrated pixel-level feature information and each first unknown semantic calibration score. Among them, the calibrated pixel-level feature information in the above-mentioned calibrated pixel-level feature information can be the calibrated pixel-level feature information. The first unknown semantic calibration score in the above-mentioned first unknown semantic calibration score can be the calibrated first unknown semantic score.

[0062] In some optional implementations of some embodiments, the above-mentioned each pixel-level feature information and each first unknown semantic score may be calibrated based on the above-mentioned each calibration pixel-level feature similarity to obtain each calibration pixel-level feature information and each first unknown semantic calibration score through the following steps:

[0063] In the first step, for each pixel-level feature information in the above-mentioned pixel-level feature information, weighted processing is performed on the above-mentioned pixel-level feature information based on the above-mentioned calibration pixel-level feature similarity by the following formula to obtain the calibration pixel-level feature information:

[0064] fc =A c ×V;

[0065] Among them, f c is the above calibration pixel-level feature information. V is the value matrix corresponding to the above pixel-level feature information.

[0066] In the second step, for each of the first unknown semantic scores, the following steps are performed:

[0067] The first sub-step is to generate a first unknown semantic score to be calibrated corresponding to the first unknown semantic score based on the first unknown semantic score. The first unknown semantic score to be calibrated may be the first unknown semantic score after maximum pooling and shape change processing. In practice, the first unknown semantic score may be subjected to maximum pooling processing by the maximum pooling technique to obtain a maximum pooling semantic score. The maximum pooling semantic score may be the first unknown semantic score after maximum pooling processing. Then, the maximum pooling semantic score may be subjected to shape change processing by a shape change function to obtain the first unknown semantic score to be calibrated. The shape change function may be a reshape function.

[0068] In the second sub-step, a preliminary calibration detection score is generated based on the calibration pixel-level feature similarity and the first unknown semantic score to be calibrated by the following formula:

[0069] S′=A c ×S m ;

[0070] Wherein, S′ is the preliminary calibration test score. Wherein, the preliminary calibration test score can be the product of the calibration pixel-level feature similarity of the first unknown semantic score to be calibrated and the calibration pixel-level feature similarity corresponding to the first unknown semantic score. m is the first unknown semantic score to be calibrated.

[0071] In the third step, for each of the generated preliminary calibration test scores, the following steps are performed:

[0072] In the first sub-step, the above preliminary calibration test score is normalized by the following formula to obtain a normalized test score:

[0073]

[0074] in, is the above normalized detection score. min is the minimum function. max is the maximum function.

[0075] In a second sub-step, the first unknown semantic score corresponding to the normalized detection score among the first unknown semantic scores is determined as the first unknown semantic score to be processed.

[0076] In the third sub-step, a first unknown semantic calibration score is generated based on the first unknown semantic score to be processed and the normalized detection score by the following formula:

[0077]

[0078] Among them, S c is the calibration score of the first unknown semantics. S is the score of the first unknown semantics to be processed. The * in can be element-wise multiplication. As an example: when When is [3,2,2,1] and S is [1,2,1,2], when multiplying element by element, Multiply the 3 in the first row and first column of S by the 1 in the first row and first column of S, and the resulting 3 is used as S c The first row and first column of the element. And so on, the final S c is [3,4,2,2].

[0079] Step 1035, for each calibrated pixel-level feature information in each calibrated pixel-level feature information, the calibrated pixel-level feature information is input into a semantically relevant decoupler and a semantically irrelevant decoupler in an open-world semantic segmentation network to perform feature decoupling on the pixel-level feature information to obtain semantically relevant feature information, a second unknown semantic score, and semantically irrelevant feature information.

[0080] In some embodiments, for each of the above-mentioned calibrated pixel-level feature information, the above-mentioned calibrated pixel-level feature information can be input into a semantically relevant decoupler and a semantically irrelevant decoupler in an open world semantic segmentation network to perform feature decoupling on the above-mentioned pixel-level feature information to obtain semantically relevant feature information, a second unknown semantic score, and semantically irrelevant feature information. Among them, the above-mentioned semantically relevant feature information can be a feature vector corresponding to a semantically relevant feature in the calibrated pixel-level feature information. The above-mentioned semantically relevant feature can be a feature vector containing key information in the calibrated pixel-level feature information. The above-mentioned key information can be information used to determine the semantic label corresponding to the pixel. The above-mentioned key information can include but is not limited to at least one of the following: shape, texture, and position. The above-mentioned semantically irrelevant feature information can be a feature vector corresponding to a semantically irrelevant feature in the calibrated pixel-level feature information. The above-mentioned semantically irrelevant feature can be a feature vector containing redundant information in the calibrated pixel-level feature information. The above-mentioned redundant information can be information irrelevant to determining the semantic label of the pixel. For example, the above-mentioned redundant information can be brightness. The second unknown semantic score can be a row of pixels in the mixed semantic image data corresponding to each row, a column of pixels in the mixed semantic image data corresponding to each column, and each element can be a matrix of probability values ​​for characterizing whether a pixel is an unknown semantic pixel. The second unknown semantic score can be a more accurate first unknown semantic score. In practice, the above-mentioned calibrated pixel-level feature information can be input into the feature decoupler included in the above-mentioned semantic correlation decoupler to obtain the semantically relevant feature information corresponding to each of the above-mentioned calibrated pixel-level feature information. In practice, the above-mentioned calibrated pixel-level feature information can be input into the classification head included in the above-mentioned semantic correlation decoupler to obtain the second unknown semantic score corresponding to each of the above-mentioned calibrated pixel-level feature information. In practice, the above-mentioned calibrated pixel-level feature information can be input into the semantically irrelevant decoupler to obtain the semantically irrelevant feature information corresponding to each of the above-mentioned calibrated pixel-level feature information.

[0081] Step 1036: Generate semantic feature information of each overlapping area based on the obtained semantically relevant feature information and the obtained semantically irrelevant feature information.

[0082] In some embodiments, each overlapping region semantic feature information can be generated based on each obtained semantically relevant feature information and each obtained semantically irrelevant feature information. Among the above-mentioned overlapping region semantic feature information, the overlapping region semantic feature information can be a feature vector obtained by splicing any semantically relevant feature information with any semantically irrelevant feature information. In practice, the above-mentioned each semantically relevant feature information and the above-mentioned each semantically irrelevant feature information can be input into the splicing function in the above-mentioned reconstruction layer to obtain each overlapping region semantic feature information.

[0083] Step 1037, performing image reconstruction processing on the semantic feature information of each overlapping area to obtain each reconstructed image data.

[0084] In some embodiments, the above-mentioned semantic feature information of each overlapping area can be subjected to image reconstruction processing to obtain each reconstructed image data. Among them, the reconstructed image data in the above-mentioned reconstructed image data can be an image obtained after the semantic feature information of the overlapping area is reconstructed. In practice, the above-mentioned semantic feature information of each overlapping area can be input into the image reconstruction network in the above-mentioned reconstruction layer to perform image reconstruction processing on the semantic feature information of each overlapping area to obtain each reconstructed image data.

[0085] Step 1038, generating a target loss value based on each first energy score matrix, each second energy score matrix, the above-mentioned first unknown semantic scores, each obtained second unknown semantic score, each reconstructed image data and semantically irrelevant feature information.

[0086] In some embodiments, a target loss value may be generated based on the first energy score matrices, the second energy score matrices, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data, and the semantically irrelevant feature information. The target loss value may be a loss value of an open world semantic segmentation network.

[0087] In some optional implementations of some embodiments, a target loss value may be generated based on the first energy score matrices, the second energy score matrices, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data and the semantically irrelevant feature information through the following steps:

[0088] The first step is to generate a reconstructed image loss value based on the above-mentioned reconstructed image data, wherein the above-mentioned reconstructed image loss value may be a loss value between the above-mentioned reconstructed image data.

[0089] In practice, for every two mixed semantic image data in the above mixed semantic image dataset, the following steps can be performed:

[0090] First, any one of the two mixed semantic image data can be determined as the first image data, and the other mixed semantic image data can be determined as the second image data.

[0091] Secondly, the first image data and the second image data may be processed as in step 1031 to step 1037 to obtain four reconstructed image data corresponding to the first image data and the second image data, wherein the four reconstructed image data are respectively expressed as: and

[0092] Among them, z re It can be semantically related feature information. un It can be semantically irrelevant feature information. It may be the semantically relevant feature information corresponding to the first image data. It may be semantically irrelevant feature information corresponding to the first image data. It may be the semantically relevant feature information corresponding to the second image data. It may be semantically irrelevant feature information corresponding to the second image data. Can be due to and The reconstructed image data. Can be due to and The reconstructed image data. Can be due to and The reconstructed image data. Can be due to and The reconstructed image data.

[0093] Then, the loss values ​​of the above four reconstructed image data can be obtained as the reconstruction loss value by the following formula:

[0094]

[0095] Among them, L recon can be the loss value of the above four reconstructed image data. 1o It may be an image area in the first image data to which the target image patch is pasted. 2o The second image data may be an image area to which the target image patch is pasted.

[0096] Finally, the sum of the obtained reconstruction loss values ​​can be determined as the above-mentioned reconstructed image loss value.

[0097] In the second step, the semantically irrelevant loss value is generated based on the above semantically irrelevant feature information through the following formula:

[0098]

[0099] Among them, L un is the semantically irrelevant loss value mentioned above. un It can be the loss value between the above semantically irrelevant feature information. i is the sequence number. E is the symbol used for averaging. is the i-th semantically irrelevant feature information among the above semantically irrelevant feature information. KL Relative entropy. N(0,1) represents a normal distribution with a mean of 0 and a variance of 1. Used for calculation The relative entropy between N(0,1).

[0100] In the third step, based on each of the first energy score matrices, the following steps are performed:

[0101] In a first sub-step, a second energy score matrix corresponding to the first energy score matrix among the second energy score matrices is determined as a target second energy score matrix.

[0102] In the second sub-step, energy score error data is generated based on the first energy score matrix and the target second energy score matrix by the following formula:

[0103] Ψ(u1,h d ,u2,h o )=|E(u1,h d )-E(u2,h o )| 2 ;

[0104] Among them, Ψ(u1,h d ,u2,h o ) is the energy fraction error data. d ,u2,h o ) may be the square of the absolute value of the difference between the first energy score matrix and the target second energy score matrix. d is the classification header in the above micro decoder. o is the classification head in the semantic segmentation network. u1 is the depth feature information corresponding to the first energy score matrix. u2 is the pixel-level feature information corresponding to the first energy score matrix. E(u1, h d ) is the first energy score matrix. E(u2,h o ) is the above target second energy score matrix.

[0105] In a third sub-step, in response to determining that the energy fraction error data satisfies a preset error condition, the energy fraction error data is determined as an energy fraction error value. The preset error condition may be that the energy fraction error data is greater than or equal to a preset error threshold. The error threshold may be a preset value. Here, the specific setting of the error threshold is not limited.

[0106] A fourth sub-step, in response to determining that the energy score error data does not satisfy the preset error condition, determines the preset error threshold as the energy score error value.

[0107] The fifth sub-step is to generate the penalty loss value based on the energy score error value using the following formula:

[0108]

[0109] Among them, L CDL It can be the penalty loss value. H is the number of rows of the first energy score matrix or the target second energy score matrix. W is the number of columns of the first energy score matrix or the target second energy score matrix. i is the index of H. j is the index of W. log is the logarithmic function with base 10. P is the probability value. h p is the classification head in the semantic correlation decoupler. y is the category label. z re It is the semantically relevant feature information corresponding to the first energy score matrix and the target second energy score matrix. For classification header p Output, z re The probability that the category label corresponding to the feature element in the i-th row and j-th column is y is the score value corresponding to the i-th row and j-th column in the second unknown semantic score corresponding to the first energy score matrix above. d is the classification header in the above micro decoder. d It is the pixel-level feature information corresponding to the first energy score matrix and the target second energy score matrix. For classification header d Output, z d The probability that the category label corresponding to the feature element in the i-th row and j-th column is y is the score value corresponding to the i-th row and j-th column in the first unknown semantic score corresponding to the first energy score matrix. d ,u2,h o ) i，j is the element value corresponding to the i-th row and j-th column in the above energy fraction error data.

[0110] In the fourth step, the sum of the above-mentioned reconstructed image loss value, the above-mentioned semantically irrelevant loss value and the generated penalty item loss values ​​is determined as the target loss value.

[0111] Step 1039, in response to determining that the target loss value satisfies the preset loss value condition, the open-world semantic segmentation network is determined as an open-world semantic segmentation model, and each first unknown semantic calibration score obtained in the last training step is determined as each target calibration score.

[0112] In some embodiments, in response to determining that the target loss value satisfies a preset loss value condition, the open-world semantic segmentation network can be determined as an open-world semantic segmentation model, and each first unknown semantic calibration score obtained in the last training step can be determined as each target calibration score. The preset loss value condition can be that the target loss value is less than a preset loss value threshold. The loss value threshold can be a pre-set value. Here, the specific setting of the loss value threshold is not limited.

[0113] Step 10310, in response to determining that the target loss value does not meet the preset loss value condition, adjusting the network parameters of the open-world semantic segmentation network, and using unused mixed semantic image data to form a mixed semantic image dataset, using the adjusted open-world semantic segmentation network as the open-world semantic segmentation network, and performing the training step again.

[0114] In some embodiments, in response to determining that the target loss value does not satisfy the preset loss value condition, the network parameters of the open-world semantic segmentation network can be adjusted, and the unused mixed semantic image data are used to form a mixed semantic image data set, and the adjusted open-world semantic segmentation network is used as the open-world semantic segmentation network to perform the training step again. In practice, the network parameters of the open-world semantic segmentation network can be adjusted using a back propagation algorithm (BP algorithm) and a gradient descent method (e.g., a mini-batch gradient descent algorithm).

[0115] Figure 2 A schematic diagram of generating various mixed semantic image data based on various known semantic image data may be provided for an open world semantic segmentation model construction method according to some embodiments of the present disclosure.

[0116] Among them, each known semantic image data may be the above-mentioned known semantic image data. Random scaling may be a process of performing random scaling processing on any one of the above-mentioned known semantic image data. Color enhancement may be a process of performing color enhancement processing on the randomly scaled known semantic image data to obtain a preprocessed sample image. Image cropping may be a process of performing image cropping on the above-mentioned preprocessed sample image based on the above-mentioned shape information to obtain each image patch. Each image patch may be the above-mentioned image patch. The sample image may be the above-mentioned sample image. Random scaling may be a process of performing random scaling on the above-mentioned sample image. Random pasting may be a process of pasting each image patch to the above-mentioned sample image to obtain a patch sample image. The patch sample image may be the above-mentioned patch sample image. “Cropping with overlapping areas” may be a process of cropping the above-mentioned patch sample image based on the above-mentioned cropping information to obtain each cropped sample image. Each cropped sample image may be the above-mentioned cropped sample image. Random masking may be a process of performing random masking on each of the above-mentioned cropped sample images to obtain each masked sample image. Each masked sample image may be the above-mentioned masked sample image. Pasting in overlapping areas may be a process of pasting the target image patches to preset positions in the mask sample images to obtain mixed semantic image data. Color jittering may be a process of adjusting brightness, saturation, hue, etc. of the mask sample images after pasting the target image patches that meet the preset pasting conditions to the preset positions in the mask sample images by using data enhancement technology.

[0117] Figure 3 A schematic diagram of calibrated pixel-level feature similarity, calibrated pixel-level feature information, and a first unknown semantic calibration score may be generated for an open-world semantic segmentation model construction method according to some embodiments of the present disclosure.

[0118] Wherein, S can represent the first unknown semantic score and the first unknown semantic score to be processed. The maximum pooling and shape change can be performed on S to obtain the first unknown semantic score to be calibrated S. m process. m The first unknown semantic score to be calibrated can be represented. S′ can represent the preliminary calibration detection score. Normalization can be performed on each S′ to obtain a normalized detection score. process. c Can be for The first unknown semantic calibration score obtained by element-wise multiplication of and S. The self-attention map A from the frozen pre-trained branch i, where the frozen pre-trained branch is the micro decoder mentioned above, A i The self-attention map output by the micro decoder can be the average of A i The process of performing average processing to obtain the pixel-level feature similarity A0 to be calibrated. α can be the above calibration coefficient. The pixel-level feature of the new branch can be the pixel-level feature information output by the above micro decoder. The shape transformation projection can be the process of performing shape transformation, projection and other processing on the pixel-level feature information to obtain the query matrix Q, the key matrix K and the value matrix V. Among them, the query matrix Q and the key matrix K can generate the unknown semantic self-attention map A through the formula n Then through A n , A0 and α to obtain the calibrated pixel-level feature similarity A c .f c Can be passed A c The calibrated pixel-level feature information obtained by and V. Matrix multiplication It can correspond to × in the formula. Add element by element It can correspond to the formula Element-wise multiplication It can correspond to the * in the formula.

[0119] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for constructing an open world semantic segmentation model, comprising: Based on the preset known semantic image data, generate each mixed semantic image data; Determining a preset number of mixed semantic image data among the mixed semantic image data as a mixed semantic image data set; Based on the mixed semantic image dataset, the following training steps are performed: Inputting the mixed semantic image dataset into a semantic segmentation network in an open world semantic segmentation network to obtain each depth feature information and each first energy score matrix; Inputting each of the deep feature information into a micro-decoder in an open-world semantic segmentation network to decode each of the mixed semantic image data to obtain each pixel-level feature information, each first unknown semantic score, each self-attention map, and each second energy score matrix; Based on the self-attention maps, generating respective calibrated pixel-level feature similarities; Based on the respective calibration pixel-level feature similarities, calibrating the respective pixel-level feature information and the respective first unknown semantic scores to obtain respective calibration pixel-level feature information and respective first unknown semantic calibration scores; For each calibrated pixel-level feature information in the respective calibrated pixel-level feature information, the calibrated pixel-level feature information is input into a semantically relevant decoupler and a semantically irrelevant decoupler in an open-world semantic segmentation network to perform feature decoupling on the calibrated pixel-level feature information to obtain semantically relevant feature information, a second unknown semantic score, and semantically irrelevant feature information; Generate semantic feature information of each overlapping area based on each obtained semantically relevant feature information and each obtained semantically irrelevant feature information; Performing image reconstruction processing on the semantic feature information of each overlapping area to obtain each reconstructed image data; Generate a target loss value based on the first energy score matrices, the second energy score matrices, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data and the semantically irrelevant feature information; In response to determining that the target loss value satisfies a preset loss value condition, determining the open-world semantic segmentation network as an open-world semantic segmentation model, and determining each first unknown semantic calibration score obtained in the last training step as each target calibration score; In response to determining that the target loss value does not satisfy the preset loss value condition, adjusting the network parameters of the open-world semantic segmentation network, using unused mixed semantic image data to form a mixed semantic image dataset, using the adjusted open-world semantic segmentation network as the open-world semantic segmentation network, and performing the training step again.

2. The method according to claim 1, wherein: The generating of each mixed semantic image data based on each preset known semantic image data includes: Randomly select a known semantic image data from each known semantic image data acquired in advance as a known semantic sample image; Preprocessing the known semantic sample image to obtain a preprocessed sample image; Based on each preset shape information, the preprocessed sample image is cropped to obtain each image patch; Randomly pasting each of the image patches to a preset sample image to obtain a patch sample image; Based on preset cropping information, the patch sample image is cropped to obtain each cropped sample image; Performing random masking on each of the cropped sample images to obtain each masked sample image; Determine each image patch that satisfies a preset shape condition among the image patches as each target image patch; For each of the mask sample images, a target image patch that meets a preset pasting condition among the target image patches is pasted to a preset position in the mask sample image to obtain mixed semantic image data.

3. The method according to claim 1, wherein: The generating each calibration pixel-level feature similarity based on each self-attention map includes: Averaging the self-attention maps to obtain the pixel-level feature similarity to be calibrated; For each pixel-level feature information in the respective pixel-level feature information, the following steps are performed: Based on the pixel-level feature information, generating a query matrix, a key matrix and a value matrix; The unknown semantic self-attention map is generated based on the query matrix, key matrix and value matrix by the following formula: Among them, A n is the unknown semantic self-attention map, softmax is the softmax function, Q is the query matrix, K T is the transpose of the key matrix, d K is the dimension of the bond matrix; The pixel-level feature similarity to be calibrated is calibrated by the following formula to generate the calibrated pixel-level feature similarity: Among them, A c is the calibration pixel-level feature similarity, α is the preset calibration coefficient, A n is the unknown semantic self-attention map, A0 is the pixel-level feature similarity to be calibrated, The * in is element-by-element multiplication. In It is element-wise addition.

4. The method according to claim 3, wherein: The step of calibrating each pixel-level feature information and each first unknown semantic score based on each calibration pixel-level feature similarity to obtain each calibration pixel-level feature information and each first unknown semantic calibration score includes: For each pixel-level feature information in the respective pixel-level feature information, weighted processing is performed on the pixel-level feature information based on the calibration pixel-level feature similarity by the following formula to obtain the calibration pixel-level feature information: f c =A c ×V; Among them, f c is the calibration pixel-level feature information, and V is the value matrix corresponding to the pixel-level feature information; For each of the first unknown semantic scores, performing the following steps: Based on the first unknown semantic score, generating a first unknown semantic score to be calibrated corresponding to the first unknown semantic score; The preliminary calibration detection score is generated based on the calibration pixel-level feature similarity and the first unknown semantic score to be calibrated by the following formula: S′=A c ×S m ; Where S′ is the preliminary calibration test score, S m is the first unknown semantic score to be calibrated; For each of the generated preliminary calibration test scores, perform the following steps: The preliminary calibration test score is normalized by the following formula to obtain a normalized test score: in, is the normalized detection score, min is the minimum function, and max is the maximum function; Determine a first unknown semantic score corresponding to the normalized detection score among the first unknown semantic scores as a first unknown semantic score to be processed; The first unknown semantic calibration score is generated based on the first unknown semantic score to be processed and the normalized detection score by the following formula: Among them, S c is the first unknown semantic calibration score, S is the first unknown semantic score to be processed, The * in the is element-wise multiplication.

5. The method according to claim 1, wherein: The generating a target loss value based on the first energy score matrices, the second energy score matrices, the first unknown semantic scores, the obtained second unknown semantic scores, the reconstructed image data and the semantically irrelevant feature information comprises: Based on the respective reconstructed image data, generating a reconstructed image loss value; The semantically irrelevant loss value is generated based on the semantically irrelevant feature information by the following formula: Among them, L un is the semantically irrelevant loss value, u is the serial number, E is the symbol used to find the mean, is the i-th semantically irrelevant feature information among the semantically irrelevant feature information, D KL represents relative entropy, N(0,1) represents a normal distribution with a mean of 0 and a variance of 1, Used for calculation Relative entropy with N(0,1); Based on each of the first energy score matrices, the following steps are performed: determining a second energy score matrix corresponding to the first energy score matrix among the second energy score matrices as a target second energy score matrix; Energy score error data is generated based on the first energy score matrix and the target second energy score matrix by the following formula: Ψ(u1,h d ,u2,h o )=|E(u1,h d )-E(u2,h o )| 2 ; Among them, Ψ(u1,h d ,u2,h o ) is the energy fraction error data, h d is the classification header in the micro decoder, h o is the classification head in the semantic segmentation network, u1 is the deep feature information corresponding to the first energy score matrix, u2 is the pixel-level feature information corresponding to the first energy score matrix, E(u1, h d ) is the first energy score matrix, E(u2,h o ) is the target second energy score matrix; In response to determining that the energy fraction error data satisfies a preset error condition, determining the energy fraction error data as an energy fraction error value; In response to determining that the energy score error data does not satisfy the preset error condition, determining a preset error threshold as an energy score error value; The penalty loss value is generated based on the energy score error value using the following formula: Wherein, H is the number of rows of the first energy score matrix or the target second energy score matrix, W is the number of columns of the first energy score matrix or the target second energy score matrix, i is the index of H, j is the index of W, log is the logarithmic function with base 10, P is the probability value, h p is the classification head in the semantic correlation decoupler, y is the category label, z re is the semantically relevant feature information corresponding to the first energy score matrix and the target second energy score matrix, For classification header p Output, z re The probability that the category label corresponding to the feature element in the i-th row and j-th column is y is the score value corresponding to the i-th row and j-th column in the second unknown semantic score corresponding to the first energy score matrix, h d is the classification head in the micro decoder, z d is the pixel-level feature information corresponding to the first energy score matrix and the target second energy score matrix, For classification header d Output, z d The probability that the category label corresponding to the feature element in the i-th row and j-th column is y is the score value corresponding to the i-th row and j-th column in the first unknown semantic score corresponding to the first energy score matrix, Ψ(u1, h d ,u2,h o ) i，j is the element value corresponding to the i-th row and j-th column in the energy fraction error data; The sum of the reconstructed image loss value, the semantically irrelevant loss value and the generated penalty item loss values ​​is determined as the target loss value.

Images