A method for establishing a visual understanding model and an image segmentation prediction method

Abstract

The application discloses a kind of visual understanding model's establishment method and image segmentation prediction method, belong to visual understanding technical field, establishment method includes: establishing conditional edge loss function;Training initial visual understanding model process is obtained target visual understanding model by adjusting model parameter through conditional edge loss function.Condition edge loss function is by sampling each anchor point corresponding positive sample and negative sample, using positive sample produces anchor point corresponding conditional local class center, by supervising conditional local class center feature so that it contains rich intraclass discriminant feature, by improving the similarity between anchor point feature and its corresponding conditional local class center feature, by the similarity between anchor point and sample based on class supervision, local class feature information around anchor point is encoded into anchor point feature.The application can make the pixel at class edge learn accurate class-related information, improve the accuracy of edge pixel classification in visual understanding task.

Description

Technical Field

[0001] This invention belongs to the field of visual understanding technology, and more specifically, relates to a method for establishing a visual understanding model and an image segmentation prediction method. Background Technology

[0002] Edge pixels refer to pixels in an image located in the region surrounding the boundary between two different categories. For visual understanding tasks such as semantic segmentation and object segmentation, current mainstream methods struggle to correctly predict the category of edge pixels. Therefore, improving the accuracy of edge pixel prediction is crucial for further enhancing the performance of visual understanding tasks, such as medical image segmentation and autonomous driving, which require the prediction of fine object contours. Because mainstream methods use stacked convolutional layers and pooling layers to reduce resolution and obtain a large receptive field, edge pixels contain many category-related features from surrounding pixels, leading to lower edge pixel classification accuracy. Furthermore, the self-attention mechanism and multi-scale feature fusion commonly used in mainstream methods further exacerbate this phenomenon. To address these issues, previous efforts to improve edge pixel accuracy mainly relied on adding auxiliary tasks for edge segmentation or supervising the inter-pixel relationships across the entire input image. However, these methods lack conditional optimization for each edge pixel, leading to conflicts with the task's optimization objectives.

[0003] The former approach optimizes the visual understanding model training process by adding an auxiliary edge pixel segmentation task to co-optimize with the target visual understanding task, encoding information about whether a pixel is an edge pixel into its features. However, this edge segmentation task simply classifies edge pixels of different semantic categories as edge pixels, while classifying pixels within objects of different semantic categories as non-edge pixels. Because it ignores the inconsistent semantic categories within edge pixels belonging to different semantic categories and within pixels belonging to different semantic categories, the edge segmentation task impairs inter-class discriminability. Furthermore, edge pixels and internal pixels belonging to the same category are also separately classified as edge and non-edge pixels, reducing consistency among pixels of the same category.

[0004] The latter method enhances the similarity among all pixels of the same category in the entire image based on pixel relationships. However, this method ignores the fact that the features of incorrectly predicted pixels during training contain noise and incorrect category-related information for the video understanding model. Supervision of pixel relationships across the entire image inevitably leads to the propagation of noise and error information from incorrectly predicted pixels across all pixels in the entire input image, thus misleading the video understanding model.

[0005] Overall, the methods described above for improving edge pixel prediction accuracy lack the ability to optimize based on the specific circumstances of each edge pixel, i.e., they lack conditional optimization, resulting in low edge pixel prediction accuracy in various computer vision understanding tasks. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for establishing a visual understanding model and an image segmentation prediction method. Its purpose is to design a conditional edge loss function. By incorporating this function into the model training process for visual understanding tasks, the model can capture more accurate and easily categorized edge pixel category-related features during training. This solves the technical problem of low edge pixel prediction accuracy in visual understanding tasks due to the lack of conditional optimization in existing edge pixel prediction accuracy improvement efforts.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for establishing a visual understanding model is provided, comprising:

[0008] S1: Establish the conditional marginal loss function;

[0009] S2: Train an initial visual understanding model using a standard dataset. The training process adjusts the model parameters using the conditional edge loss function to obtain the target visual understanding model. Each standard dataset includes: an input image and its corresponding ground truth label.

[0010] The conditional edge loss function includes:

[0011] The accuracy-based conditional sampling module is used to determine multiple candidate samples corresponding to each anchor point in a preset region of the feature map of the input map, and to determine a set of positive samples and a set of negative samples from them; the ground truth labels of the anchor points, the ground truth labels of the candidate samples, and the segmentation predictions of the candidate samples are consistent.

[0012] The conditional local class center generation module takes the samples in the positive sample set corresponding to each anchor point as its input to generate the conditional local class center features of each anchor point, which are used to characterize the surrounding local information and category-related information of each anchor point.

[0013] The conditional local class center classification loss structure is used to enhance the class-related information between the local information and the ground value class of the conditional local class center features corresponding to each anchor point in the positive sample set.

[0014] Anchor-conditional local class center similarity loss structure is used to enhance the feature similarity between each anchor and its corresponding conditional local class center;

[0015] Anchor-sample similarity loss structure is used to encode the fine-grained category relationship information between each anchor and its corresponding positive and negative samples into the anchor features;

[0016] The final output loss of the conditional margin loss function is the sum of the output losses of the conditional local class center classification loss structure, the anchor-conditional local class center similarity loss structure, and the anchor-sample similarity loss structure.

[0017] In one embodiment, the accuracy-based conditional sampling module includes:

[0018] The local neighbor sampling unit is used to select candidate samples within a window centered on each anchor point.

[0019] The correct classification sampling unit is used to ensure that the candidate samples selected for each anchor point can be correctly classified by the visual understanding model currently being trained.

[0020] The same-category positive sample sampling unit is used to select positive samples from the candidate samples corresponding to each anchor point that satisfy the following: the true value label category is the same as that of the corresponding anchor point, so as to form the positive sample set;

[0021] The non-same-class negative sample sampling unit is used to select negative samples from the candidate samples corresponding to each anchor point that satisfy the following condition: the true value label category of the corresponding anchor point is different from that of the negative sample set.

[0022] In one embodiment, the conditional local class center generation module is used to take the set of positive samples corresponding to the anchor point as input, perform a channel-wise averaging operation on the features, and output the averaged features as the conditional local class center features corresponding to the anchor point.

[0023] In one embodiment, the conditional local class center classification loss structure includes:

[0024] A parameter-shared model segmentation head is used to obtain the segmentation prediction of the conditional local class center. The segmentation head and parameters are structurally consistent with the segmentation head in the semantic segmentation network model, and the category to which it belongs is predicted based on the features of the conditional local class center.

[0025] The cross-entropy loss structure for conditional local class center prediction is used to supervise that the prediction generated by the conditional local class center through the parameter-shared segmentation head is still the ground truth class of the anchor corresponding to the conditional local class center, so that it contains rich class-related features of the ground truth class of the anchor.

[0026] In one embodiment, the anchor-conditional local class center similarity loss structure includes:

[0027] The feature mean squared error loss structure is used to constrain anchor features to move closer to their corresponding conditional local class center features, thereby increasing the similarity between the two in the feature space and encoding richer category-related information into the anchor features. It also uses the rich local context information in the conditional local class center features to reduce the difficulty of mean squared error loss optimization.

[0028] The gradient cut-off operation unit is used to maintain the tendency of anchor features to move closer to conditional local class center features during the training of visual understanding models, and to prevent the generated conditional local class centers from changing their tendency to move closer to anchor features due to the constraints of the feature mean squared error loss structure.

[0029] In one embodiment, the anchor-sample similarity loss structure includes:

[0030] Anchor-sample similarity calculation structure, used to calculate the cosine similarity between anchor features and sample features, characterizing the degree of similarity between anchors and samples in the feature space;

[0031] A similarity truth value generation unit is used to generate truth value labels in the similarity mean squared error loss structure. The truth value labels are generated based on whether the anchor point and the sample have the same truth value category label.

[0032] The similarity mean squared error loss structure is used to increase the consistency between the anchor point and positive samples in the feature space, and reduce the consistency between the anchor point and negative samples in the feature space. This encodes the fine-grained local category feature information around the anchor point into the anchor point feature, making it easier to classify correctly.

[0033] According to another aspect of the present invention, an image segmentation prediction method is provided, comprising: inputting an image to be processed into the target visual understanding model to obtain a segmentation prediction result of the image to be processed.

[0034] According to another aspect of the present invention, an image recognition method is provided, comprising: obtaining a corresponding segmentation prediction result according to the image segmentation prediction method; performing scene understanding on the segmentation prediction result of the image to be processed to obtain an image recognition result of the image to be processed.

[0035] According to another aspect of the present invention, an image recognition device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method.

[0036] According to another aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0037] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0038] (1) The visual understanding model building method provided by this invention, by adding a conditional edge loss function during the model training process for performing visual understanding tasks, enables the model to capture more accurate and easily classifiable edge pixel category-related features during the model training process for visual understanding tasks. This allows for the improvement of edge pixel prediction accuracy in visual understanding tasks by utilizing these edge pixel category-related features. Compared to existing technologies that only use the conventional cross-entropy loss function, we propose that the conditional edge loss function and the cross-entropy loss function jointly adjust the model parameters, achieving a significant advantage in edge pixel prediction accuracy.

[0039] (2) The present invention provides a conditional sampling module based on accuracy in the conditional edge loss function. By sampling based on accuracy and limiting the sampling of spatial distance from the anchor point, error category related information and noise in the sample are filtered out, and the influence of local context information that is different from the anchor point in the sample with too far spatial distance is filtered out, thereby improving the quality of edge pixel features.

[0040] (3) The present invention provides a conditional local class center classification loss structure in the conditional edge loss function. By enhancing the similarity between the anchor point and the conditionally generated conditional local class center, and the fact that the conditional local class center features contain contextual information unique to the anchor point, the similarity constraint is easier to optimize. This makes the anchor point features contain richer category-related information, enhances the similarity between the anchor point features and the class-specific discriminative features, and improves the accuracy of edge pixels in computer vision understanding tasks.

[0041] (4) The present invention provides an anchor-sample similarity loss structure in the method for establishing ground conditional edge loss function. By supervising the feature similarity between the anchor and the positive and negative samples obtained by conditional sampling, fine-grained local category information is encoded into the anchor features, which enhances the intra-class consistency between the anchor and local pixels of the same class and improves the inter-class distinguishability between the anchor and local pixels of different classes. Attached Figure Description

[0042] Figure 1 A schematic diagram of the conditional edge loss function provided in an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a conditional sampling module based on accuracy provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the conditional local class center classification loss structure provided in an embodiment of the present invention;

[0045] Figure 4This is a schematic diagram of the anchor-conditional local class center similarity loss structure provided in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the anchor-sample similarity loss structure provided in an embodiment of the present invention;

[0047] Figures 6a to 6d Visualization results of the prediction results of the baseline model and the model of this invention in the geographic information system;

[0048] Figures 7a to 7d This is a visualization of the error pixels in the prediction results of the baseline model and the model of this invention in an autonomous driving scenario. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0050] Example 1:

[0051] This invention provides a method for establishing a visual understanding model, comprising:

[0052] In the backbone network used to perform the target visual understanding task, the conditional edge loss function obtained by the conditional edge loss function establishment method provided by this invention is inserted into the training stage of the visual understanding model. It is optimized together with the other loss functions of the original visual understanding model to train the visual understanding model.

[0053] The visual understanding model is trained using a standard training set to obtain a well-trained visual understanding model.

[0054] In the standard training set, each sample consists of an image related to the target visual understanding task and its corresponding ground truth label, which is used to indicate the task result.

[0055] In this embodiment, after the conditional edge loss function is inserted into the training phase, the anchor set of the input conditional edge loss function is obtained from the feature image output after the original image features are sequentially passed through the modules in the backbone network before the insertion position.

[0056] The specific output features of which module in the backbone network are used can be set according to the actual requirements of the target visual understanding task; the features of the anchor point can be pixel features or image raster features.

[0057] The method for establishing the conditional marginal loss function includes:

[0058] A conditional edge loss function is established to supervise the class-related information of edge pixels (i.e. anchor points) in the input image during the training phase of the visual understanding model. This enhances the intra-class consistency between anchor points and local pixels of the same class, improves the inter-class distinguishability between anchor points and local pixels of different classes, and improves the edge pixel prediction accuracy without adding any additional inference cost during the inference phase of the visual understanding model.

[0059] like Figure 1 As shown, the conditional margin loss function includes: a conditional sampling module based on accuracy, a conditional local class center generation module, a conditional local class center classification loss structure, an anchor-conditional local class center similarity loss structure, and an anchor-sample similarity loss structure.

[0060] The accuracy-based conditional sampling module is used to sample the positive and negative samples corresponding to each anchor point (i.e., edge pixels) as input for subsequent modules. During the sampling process, different positive and negative samples are selected for each anchor point, and the spatial distance and whether the predicted category of the candidate sample by the current visual understanding model is consistent with the ground truth label are considered.

[0061] The conditional local class center generation module is used to generate conditional local class center features that simultaneously possess unique local information and category-related information around each anchor point. Each anchor point corresponds to a unique conditional local class center feature, which has more local context information around its corresponding anchor point compared to the global class center in the entire input image.

[0062] The Conditional Local Class Center (CLC) classification loss structure is used to enhance the local information and class-related information of the ground truth class in the CLC features corresponding to each anchor point, thereby improving the accuracy of the CLC features under the classification of the model segmentation head, and ensuring that the generated CLC features are still in the feature space of the pixel features in the feature map.

[0063] The anchor-conditional local class center similarity loss structure is used to enhance the feature similarity between each anchor and its corresponding conditional local class center, thereby enhancing the class-related information of the ground truth class in each anchor feature. The local context information in the conditional local class center features makes the similarity loss between it and its corresponding anchor easier to optimize.

[0064] Anchor-sample similarity loss structure is used to encode the local, fine-grained anchor-sample class relationship of each anchor into the anchor feature, thereby reducing the feature similarity between the anchor and its negative samples and increasing the feature similarity between the anchor and its positive samples.

[0065] Wherein, anchor points and samples are pixels or image grids, and the final output loss of the conditional edge loss function is the sum of the output losses of the conditional local class center classification loss structure, the anchor point-conditional local class center similarity loss structure, and the anchor point-sample similarity loss structure.

[0066] The conditional margin loss function provided in this embodiment ultimately outputs a loss that is the sum of the output losses of the conditional local class center classification loss structure, the anchor-conditional local class center similarity loss structure, and the anchor-sample similarity loss structure, specifically:

[0067] L cbl =L cce +αL feat +βL simi ,

[0068] Where L cbl This is the Conditional Boundary Loss function, where α and β are manually set coefficients used to balance the output losses. α and β can be arbitrarily adjusted according to the visual understanding task and the backbone network; empirically, they can be set to α = 10 and β = 5.

[0069] Example 2

[0070] The accuracy-based conditional sampling module provided in this embodiment includes:

[0071] The local neighbor sampling unit is used to select candidate samples for each anchor point within a window centered on the anchor point.

[0072] The correct classification sampling unit is used to select candidate samples for each anchor point that can be correctly classified by the visual understanding model currently being trained.

[0073] The same-category positive sample sampling unit is used to select positive samples from the candidate samples corresponding to each anchor point, which satisfy the following condition: the true value label category is the same as that of the corresponding anchor point, so as to form a positive sample set.

[0074] The non-same-class negative sample sampling unit is used to select negative samples from the candidate samples corresponding to each anchor point, which must satisfy the following condition: the negative sample is not the same as the true value label category of its corresponding anchor point, so as to form a negative sample set.

[0075] In this embodiment, by sampling based on accuracy and limiting sampling to the spatial distance of the anchor point, incorrect category-related information and noise in the samples are filtered out. The influence of local contextual information different from the anchor point in samples with excessively large spatial distances is also filtered out, improving the quality of edge pixel features. Simultaneously, by enhancing the similarity between the anchor point and the conditionally generated conditional local class centers, and considering that the conditional local class center features contain contextual information specific to the anchor point's surroundings, the similarity constraint is easier to optimize. This results in richer category-related information in the anchor point features, enhancing the similarity between the anchor point features and the class-specific discriminative features, and improving the accuracy of edge pixels in computer vision understanding tasks. Furthermore, by supervising the feature similarity between the anchor point and the positive and negative samples obtained from conditional sampling, fine-grained local category-related information is encoded into the anchor point features, enhancing the intra-class consistency between the anchor point and local pixels of the same class, and improving the inter-class discriminability between the anchor point and local pixels of different classes.

[0076] The conditional edge loss function provided in this embodiment includes a sampling strategy for the accuracy-based sampling module, specifically as follows: Figure 2 As shown in the example. Figure 2 In the diagram, each square represents a pixel or an image grid in a feature map. a, b, c, d, e, f, g, and h represent different pixels or image grids, and different colors represent different meanings, such as... Figure 2 As shown. In the field of visual understanding, the boundary between different categories of truth labels is considered an edge. If an edge appears in a certain neighborhood of a pixel or image raster, that pixel is considered an edge pixel (i.e., an anchor point). For example, when an edge pixel is defined as an edge appearing in an 8-neighborhood, a, b, c, d, e, f, g, h and the current anchor point are all edge pixels, while the top-left pixel or image raster is not an edge pixel.

[0077] like Figure 2 As shown in this embodiment, the strategy of the local neighbor sampling unit is: only when other pixels or image grids are within the neighborhood of the current anchor point can they be sampled as candidate samples.

[0078] As an optional implementation, in this embodiment, the neighborhood range of the anchor point is an 8-neighborhood. Any other neighborhood range, such as a 4-neighborhood, can be used instead. When using an 8-neighborhood, the candidate samples after the current anchor point passes through the local neighbor sampling unit are a, b, c, d, e, f, g, and h.

[0079] The correct classification sampling unit will resample the candidate samples obtained by the local neighbor sampling unit. Its sampling strategy is: only when other pixels or image rasters are correctly predicted by the current visual understanding model, that is, when the category predicted by the model is the same as the ground truth label, can they be sampled as candidate samples.

[0080] The same-class positive sample sampling unit will resample the candidate samples obtained by the local neighbor sampling unit and the correct classification sampling unit to obtain positive samples. Its sampling strategy is: only when other pixels or image grids have the same ground truth label category as the current anchor point can they be sampled as positive samples.

[0081] The non-same-class negative sample sampling unit will resample the candidate samples obtained by the local neighbor sampling unit and the correct classification sampling unit to obtain negative samples. Its sampling strategy is: only when other pixels or image grids are not the same as the ground truth label category of the current anchor point can they be sampled as negative samples.

[0082] After sampling, the positive samples obtained form the positive sample set of the current anchor point, and the negative samples obtained form the negative sample set of the current anchor point, such as... Figure 2 As shown, the positive sample set of the current anchor point is {a,d,f,g}, and the negative sample set of the current anchor point is {c,e}.

[0083] Example 3

[0084] The conditional edge loss function provided in this embodiment, and the conditional local class center generation module provided in this example are simple mean conditional local class center generation modules, are operated as follows:

[0085]

[0086] Where e i The conditional local class center feature corresponding to anchor point i. Let i be the number of positive samples at anchor point i. Let i be the feature of the j-th positive sample at anchor point i.

[0087] Example 4

[0088] The conditional local class center classification loss structure in the conditional edge loss function provided in this embodiment is as follows: Figure 3 As shown in the image. Figure 3 In the diagram, e1 to e4 represent the conditional local class center features corresponding to each anchor point. The structure and parameters of the parameter-sharing model segmentation head are completely identical to those of the visual understanding model.

[0089] The parameter-shared model segmentation head takes the conditional local class center features of each anchor point as input, performs class prediction, and outputs the conditional local class center segmentation prediction, resulting in a segmentation prediction map (logits).

[0090] The cross-entropy loss structure for conditional local class center prediction takes the conditional local class center segmentation prediction and ground truth labels as input, and outputs the calculated loss. The goal is to ensure that the predicted conditional local class centers for each anchor point are consistent with the ground truth labels for that anchor point. The specific operation of the cross-entropy loss structure for conditional local class center prediction provided in this example is as follows:

[0091]

[0092] Where L cce Let CE(·,·) be the output loss of the cross-entropy loss structure predicted by conditional local class centers, and let CE(·,·) be the cross-entropy loss function. Let y be the predicted vector of the conditional local class center corresponding to the i-th anchor point. i Let be the truth label for the i-th anchor point.

[0093] Example 5

[0094] The conditional edge loss function provided in this embodiment includes an anchor-conditional local class center similarity loss structure, specifically as follows: Figure 4 As shown.

[0095] The gradient detachment operation takes the conditional local class centers of each anchor point as input and outputs identical data, but it cannot be passed back to the original conditional local class centers of each anchor point during backpropagation of the visual understanding model. For example, the "detach" function can be used to implement this operation in the PyTorch framework.

[0096] The feature mean squared error loss structure takes the conditional local class center features of each anchor point (after gradient clipping) and the anchor point features as input, constrains the anchor point features to be as similar as possible to their corresponding conditional local class center features in the feature space, and outputs the features. Furthermore, if the number of positive samples corresponding to an anchor point is less than 1, that anchor point is not included in the calculation of the feature mean squared error loss structure. The specific operation of the feature mean squared error loss structure provided in this example is as follows:

[0097]

[0098] Where L feat The output loss is the feature mean squared error, where N is the total number of anchor points involved in the calculation, and f i This is the feature of the anchor point.

[0099] Example 6

[0100] The conditional margin loss function provided in this embodiment includes an anchor-sample similarity loss structure as follows: Figure 5As shown, the anchor-sample similarity calculation structure takes the features of each anchor point, the negative sample features corresponding to each anchor point, and the positive sample features corresponding to each anchor point as input, and calculates the cosine similarity between the anchor point features and all samples corresponding to them. The specific operation of the anchor-sample similarity calculation structure is as follows:

[0101]

[0102] in The similarity is calculated between anchor point i and its j-th sample.

[0103] The similarity ground truth generation unit takes ground truth labels as input and outputs the ground truth similarity between each anchor point and its corresponding sample, based on whether the ground truth features of the sample and the anchor point are the same (i.e., whether the sample is a positive sample). This supervises the calculation of anchor point-sample similarity. The specific operation of the similarity ground truth generation unit provided in this example is as follows:

[0104]

[0105] Among them, a ij Let i be the ground truth similarity between anchor point i and the j-th sample. Let i be the set of positive samples. Let i be the set of negative samples for anchor point i.

[0106] The similarity mean squared error loss structure takes anchor-sample similarity and ground truth similarity as input, constrains the anchor-sample similarity to be close to the ground truth similarity, and outputs the loss. The specific operation of the similarity mean squared error loss structure provided in this example is as follows:

[0107]

[0108] Among them, L simi The output loss is the similarity mean squared error loss structure. denoted as the number of negative samples at anchor point i.

[0109] Example 7

[0110] This embodiment provides an image segmentation prediction method, which includes: inputting the image to be processed into a target visual understanding model to obtain the segmentation prediction result of the image to be processed.

[0111] Example 8

[0112] This invention provides an image recognition method, comprising: obtaining a corresponding segmentation prediction result based on an image segmentation prediction method; performing scene understanding on the segmentation prediction result of the image to be processed to obtain the image recognition result of the image to be processed.

[0113] Specifically, the visual understanding task execution method includes: inputting the image of the visual understanding task to be performed into a trained visual understanding model to obtain the task result. The trained visual understanding model is obtained by the visual understanding model establishment method provided in the above embodiments.

[0114] Example 9

[0115] This embodiment provides an image recognition device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method.

[0116] Example 10

[0117] This embodiment discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described.

[0118] The following explanation, in conjunction with some specific application scenarios, further illustrates the beneficial effects that this invention can achieve:

[0119] The conditional edge loss function first determines whether a pixel is an edge in the feature map output by the backbone network, using the presence of edges within a 5x5 neighborhood as the standard, thus obtaining an anchor point set. A conditional sampling module based on accuracy samples from these anchor points, obtaining positive and negative sample sets for each anchor point. Anchor-sample loss is then used to calculate the loss. Simultaneously, a simple mean-based conditional local class center generation module generates conditional local class centers by averaging the positive sample sets of the anchor points. The conditional local class center classification loss and anchor-conditional local class center similarity loss are then calculated. Finally, the anchor-sample loss and anchor-conditional local class center similarity loss are multiplied by their respective coefficients (α = 10, β = 5), and then added to the conditional local class center classification loss to obtain the conditional edge loss function.

[0120] Using semantic segmentation as the target visual understanding task, ResNet101 is employed as the backbone network for this task, and OCRNet as the decoder. The conditional edge loss function established in Example 1 is inserted into the training process of the ResNet101-OCRNet model to conditionally constrain edge pixel features and improve edge pixel prediction accuracy. After inserting the conditional edge loss function during the ResNet-OCRNet training process, the visual understanding task model is obtained.

[0121] The initial visual understanding model was trained using the ADE20K dataset. The ADE20K dataset is an image segmentation dataset for complex scenes, containing 20,000 images as the training set, 2,000 images as the validation set, and 3,000 images as the test set. Each pixel is labeled with a predefined category, and the dataset has 150 predefined semantic labels. The training set was randomly divided into training subsets of the same size, specifically, each training subset was 16 pixels in size. Data augmentation was performed on the data in each training subset to improve accuracy. The data augmentation methods included the following operations: (1) Calculating the mean of each channel of the images in the training set. (2) Subtracting the mean of each image from the image in the training subset. (3) Randomly flipping horizontally and randomly scaling the images, with scaling factors of any one of {0.5, 0.75, 1.0, 1.5, 1.7}.

[0122] During training, one training subset is trained at a time, and the completion of training for all training subsets constitutes one iteration. Training is repeated until the maximum number of iterations is reached, resulting in a trained visual understanding model. In practice, a maximum of 100,000 iterations is preferred. The training process in one iteration is as follows: the network parameters of the visual understanding model are trained using forward and backpropagation algorithms. Forward propagation calculates the loss function for each training subset, and backpropagation obtains the gradient for that subset. Cross-entropy loss is used for loss calculation.

[0123] The trained visual understanding model was used to perform object segmentation tasks in different scenarios. The visualization results of the segmentation predictions of the visual understanding model in different scenarios are shown in the figure. Among them, the results in the geographic information system are shown in the figure. Figures 6a to 6d As shown, the results in the autonomous driving scenario are as follows: Figures 7a to 7d As shown. Figure 6a For the input image, Figure 6b These are truth value labels; different shades of color represent different categories of truth values. Figure 6c In ~d, different shades of color represent different category predictions. Figure 6c This represents the baseline model prediction results, i.e., the segmentation prediction results of the visual understanding model that does not use the conditional marginal loss function during training. Figure 6d The segmentation prediction results of the visual understanding model obtained according to the above embodiments are shown. Figures 6a to 6d In the results, compared to the baseline model, the visual understanding model using the conditional edge loss function shows more accurate category edge predictions for the swimming pool handrail, window edges, and the left-hand pole in the image. Figures 7a to 7d middle, Figure 7a For the input image, Figure 7b These are truth value labels; different shades of color represent different categories of truth values. Figure 7cIn Figure d, the non-white parts are pixels that were predicted incorrectly, and different colors represent the truth label category of the incorrect pixels. The white parts are pixels that were predicted correctly. Figure 7c This is a graph showing the prediction errors of the baseline model. Figure 7d This is a prediction error map of the visual understanding model obtained in the above embodiments. Figures 7a to 7d Compared to the baseline model, the visual understanding model using the conditional edge loss function significantly reduces the number of erroneous pixels in the prediction results, and shows a significant improvement in edge prediction performance for predicting cars, utility poles, traffic lights, roadblocks, etc.

[0124] In summary, this invention can conditionally supervise the features of edge pixels, obtain accurate category-related information of edge pixels, and effectively improve the prediction accuracy of edge pixels in visual understanding tasks. This invention can be applied to fields such as autonomous driving, geographic information systems, video surveillance, medical image analysis, and robotics to accurately perform visual understanding tasks.

[0125] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for establishing a visual understanding model, characterized in that, include: S1: Establish a conditional margin loss function; the conditional margin loss function includes: a conditional sampling module based on accuracy, a conditional local class center generation module, a conditional local class center classification loss structure, an anchor-conditional local class center similarity loss structure, and an anchor-sample similarity loss structure; S2: Train an initial visual understanding model using a standard dataset. The training process adjusts the model parameters using the conditional edge loss function to obtain the target visual understanding model. Each standard dataset includes: an input image and its corresponding ground truth label. The accuracy-based conditional sampling module is used to determine multiple candidate samples corresponding to each anchor point within a preset region of the feature map of the input image, and to determine a positive sample set and a negative sample set from them; the ground truth labels of the anchor points, the ground truth labels of the candidate samples, and the segmentation predictions of the candidate samples are consistent; the conditional local class center generation module takes the samples in the positive sample set corresponding to each anchor point as its input, and is used to generate conditional local class center features for each anchor point, which are used to characterize the surrounding local information and category-related information of each anchor point; the conditional local class center classification loss structure is used to enhance the positive sample set. The local information in the conditional local class center features corresponding to each anchor point is related to the category of its ground truth class; the anchor-conditional local class center similarity loss structure is used to enhance the feature similarity between each anchor point and its corresponding conditional local class center; the anchor-sample similarity loss structure is used to encode the fine-grained category relationship information between each anchor point and its corresponding positive and negative samples into the anchor point features; the final output loss of the conditional margin loss function is the sum of the output losses of the conditional local class center classification loss structure, the anchor-conditional local class center similarity loss structure, and the anchor-sample similarity loss structure.

2. The method for establishing a visual understanding model as described in claim 1, characterized in that, The accuracy-based conditional sampling module includes: The local neighbor sampling unit is used to select candidate samples within a window centered on each anchor point. The correct classification sampling unit is used to ensure that the candidate samples selected for each anchor point can be correctly classified by the visual understanding model currently being trained. The same-category positive sample sampling unit is used to select positive samples from the candidate samples corresponding to each anchor point that satisfy the following: the true value label category is the same as that of the corresponding anchor point, so as to form the positive sample set; The non-same-class negative sample sampling unit is used to select negative samples from the candidate samples corresponding to each anchor point that satisfy the following condition: the true value label category of the corresponding anchor point is different from that of the negative sample set.

3. The method for establishing a visual understanding model as described in claim 2, characterized in that, The conditional local class center generation module is used to take the set of positive samples corresponding to the anchor point as input, perform a channel-wise averaging operation on the features, and output the averaged features as the conditional local class center features corresponding to the anchor point.

4. The method for establishing a visual understanding model as described in claim 2, characterized in that, The conditional local class center classification loss structure includes: A parameter-shared model segmentation head is used to obtain the segmentation prediction of the conditional local class center. The segmentation head and parameters are structurally consistent with the segmentation head in the semantic segmentation network model, and the category to which it belongs is predicted based on the features of the conditional local class center. The cross-entropy loss structure for conditional local class center prediction is used to supervise that the prediction generated by the conditional local class center through the parameter-shared segmentation head is still the ground truth class of the anchor corresponding to the conditional local class center, so that it contains rich class-related features of the ground truth class of the anchor.

5. The method for establishing a visual understanding model as described in claim 1, characterized in that, The anchor-conditional local class center similarity loss structure includes: The feature mean squared error loss structure is used to constrain anchor features to move closer to their corresponding conditional local class center features, thereby increasing the similarity between the two in the feature space and encoding richer category-related information into the anchor features. It also uses the rich local context information in the conditional local class center features to reduce the difficulty of mean squared error loss optimization. The gradient cut-off operation unit is used to maintain the tendency of anchor features to move closer to conditional local class center features during the training of visual understanding models, and to prevent the generated conditional local class centers from changing their tendency to move closer to anchor features due to the constraints of the feature mean squared error loss structure.

6. The method for establishing a visual understanding model as described in claim 1, characterized in that, The anchor-sample similarity loss structure includes: Anchor-sample similarity calculation structure, used to calculate the cosine similarity between anchor features and sample features, characterizing the degree of similarity between anchors and samples in the feature space; A similarity truth value generation unit is used to generate truth value labels in the similarity mean squared error loss structure. The truth value labels are generated based on whether the anchor point and the sample have the same truth value category label. The similarity mean squared error loss structure is used to increase the consistency between the anchor point and positive samples in the feature space, and reduce the consistency between the anchor point and negative samples in the feature space. This encodes the fine-grained local category feature information around the anchor point into the anchor point feature, making it easier to classify correctly.

7. An image segmentation prediction method, characterized in that, include: The image to be processed is input into the target visual understanding model according to any one of claims 1-6 to obtain the segmentation prediction result of the image to be processed.

8. An image recognition method, characterized in that, include: The image segmentation prediction method according to claim 7 obtains the corresponding segmentation prediction result; Scene understanding is performed on the segmentation prediction results of the image to be processed to obtain the image recognition results of the image to be processed.

9. An image recognition device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method of claim 8.

Images