Scene semantic segmentation model training method and device, and electronic device
By training the model using an adversarial mechanism, the problem of misclassification of unseen categories in scene semantic segmentation algorithms is solved, improving the scene understanding accuracy of autonomous vehicles and ensuring correct planning decisions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NEUSOFT REACH AUTOMOBILE TECH (SHENYANG) CO LTD
- Filing Date
- 2022-12-30
- Publication Date
- 2026-07-10
AI Technical Summary
Existing scene semantic segmentation algorithms may misclassify unseen categories during model training, leading to incorrect understanding of the scene by autonomous vehicles and an inability to make correct planning decisions.
The model is trained using an adversarial mechanism. By adding a classification margin to the softmax cross-entropy loss function, the distance between each category in the high-dimensional space is increased, the inter-class distance is increased, the probability of missegmentation is reduced, and unknown objects are classified as the else category.
It effectively improves the classification accuracy of known categories and the separation between unknown and known categories, reduces the probability of incorrect segmentation, and ensures that autonomous vehicles can accurately understand the scene.
Smart Images
Figure CN115953778B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of model training, and in particular to a method, apparatus, and electronic device for training a scene semantic segmentation model. Background Technology
[0002] Scene semantic segmentation plays a very important role in autonomous driving systems. Driving environments are generally open environments, meaning that the scenes are quite complex and many types of objects have never appeared during model training. Therefore, scene semantic segmentation in autonomous driving systems is an open-set problem.
[0003] Existing scene semantic segmentation algorithms may misclassify some categories that have not been seen during training, leading to incorrect understanding of the scene by autonomous vehicles and thus preventing them from making better planning decisions. Summary of the Invention
[0004] In view of this, the purpose of the present invention is to provide a method, apparatus and electronic device for training a scene semantic segmentation model, so as to alleviate the technical problem of being unable to make planning decisions based on the correct scene category classification.
[0005] In a first aspect, the embodiment provides a method for training a scene semantic segmentation model, including:
[0006] Determine training samples and an initial scene semantic segmentation model. The training samples include sample images and labels corresponding to the categories of pixels in the sample images. The categories include multiple known categories and other categories. The scene semantic segmentation model includes contrast features corresponding to each known category and contrast features corresponding to other categories.
[0007] The scene semantic segmentation model extracts features from the target sample image to obtain target semantic features and determines the target category of the target semantic features;
[0008] Based on the target category, the category label of the target sample image, and the opposite point features, the parameters of the initial scene semantic segmentation model are optimized according to the preset expectation to determine the trained scene semantic segmentation model.
[0009] The preset expectation includes that the first distance between the target semantic feature and the opposite feature corresponding to the target category is greater than the second distance between the target semantic feature and other semantic features in the target category; the target category is the same as the label category of the target sample image.
[0010] In an optional implementation, it further includes:
[0011] The opposing features of the target category are optimized based on all semantic features of the target category.
[0012] In an optional implementation, optimizing the contrast features of the target category based on all semantic features of the target category includes:
[0013] Determine the third distance between all semantic features of the target category and the opposite feature of the target category;
[0014] Determine the fourth distance between all semantic features of other categories and the opposite feature of the target category;
[0015] The opposing features of the target category are optimized based on a pre-defined optimization algorithm so that the sum of all the third distances approaches infinity and the sum of all the fourth distances approaches infinity.
[0016] In an optional implementation, the scene semantic segmentation model extracts features from the target sample image to obtain target semantic features, including:
[0017] The scene semantic segmentation model extracts multi-level semantic features from the target sample image;
[0018] The semantic features extracted from the feature image of the previous layer in the multi-level semantic features are determined as the target semantic features.
[0019] In an optional implementation, each pixel corresponds to a category label, an opposite feature, a target semantic feature, and a target category.
[0020] In an optional implementation, the scene semantic segmentation model includes a feature extraction layer and a classification layer, wherein the feature extraction layer includes the contrast features.
[0021] In an optional implementation, the preset expectation further includes reaching a desired value based on the cross-entropy loss function and the classification interval, where the classification interval is the interval between the boundaries corresponding to the categories.
[0022] Secondly, an embodiment provides a scene semantic segmentation model training device, the device comprising:
[0023] The first determining module determines training samples and an initial scene semantic segmentation model. The training samples include sample images and labels corresponding to the categories of pixels in the sample images. The categories include multiple known categories and other categories. The scene semantic segmentation model includes contrast features corresponding to each known category, and the contrast features correspond to other categories.
[0024] The second determining module involves the scene semantic segmentation model extracting features from the target sample image to obtain target semantic features and determining the target category of the target semantic features.
[0025] The training module optimizes the parameters of the initial scene semantic segmentation model according to a preset expectation based on the target category, the category label of the target sample image, and the opposite point features, and determines the trained scene semantic segmentation model.
[0026] The preset expectation includes that the first distance between the target semantic feature and the opposite feature corresponding to the target category is greater than the second distance between the target semantic feature and other semantic features in the target category; the target category is the same as the label category of the target sample image.
[0027] Thirdly, an embodiment provides an electronic device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method described in any of the foregoing embodiments.
[0028] Fourthly, an embodiment provides a machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the steps of the method described in any of the foregoing embodiments.
[0029] The present invention provides a scene semantic segmentation model training method, apparatus and electronic device. By training a model with an adversarial mechanism and adding a classification margin to the original softmax cross-entropy loss function, it can maximize the distance between different categories in high-dimensional space, increase the distance between classes, reduce the probability of missegmentation between categories, and at the same time make unknown objects as much as possible into the else category without interfering with the segmentation of known meaningful categories of objects.
[0030] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.
[0031] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0032] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0033] Figure 1 A flowchart of a scene semantic segmentation model training method provided in an embodiment of the present invention;
[0034] Figure 2 This is a schematic diagram of a scene semantic segmentation model structure provided in an embodiment of the present invention;
[0035] Figure 3 A functional block diagram of a scene semantic segmentation model training device provided in an embodiment of the present invention;
[0036] Figure 4 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] Typically, objects in a scene image that have a significant impact on driving, such as cars, people, curbs, roads, trees, and buildings, are assigned specific labels. Other objects that do not belong to the above categories are labeled as the else category and are classified as a single class during training.
[0039] However, during testing, there are still some object categories that were not seen during training. The model cannot accurately classify them into the else class, but may classify them into other specific categories of great significance. This leads to the autonomous vehicle having an incorrect understanding of the scene, thus failing to make better planning decisions.
[0040] Based on this, embodiments of the present invention provide a scene semantic segmentation model training method, apparatus, and electronic device. Most existing deep learning-based scene understanding algorithms focus on improving the classification accuracy of known categories and the fit of object edges, but do not consider unknown categories in driving scenes. Embodiments of the present invention propose an adversarial mechanism training method that considers both the classification accuracy of known categories and the separability between the unknown and known categories in the space.
[0041] To facilitate understanding of this embodiment, a scene semantic segmentation model training method disclosed in this embodiment of the invention will first be described in detail.
[0042] Figure 1 A flowchart of a scene semantic segmentation model training method provided in an embodiment of the present invention.
[0043] like Figure 1 As shown, the method includes the following steps:
[0044] S102, determine the training samples and the initial scene semantic segmentation model. The training samples include sample images and labels corresponding to the categories of the pixels in the sample images. The categories include multiple known categories and other categories. The scene semantic segmentation model includes the opposite point features corresponding to each known category. The opposite point features are closer to the other categories.
[0045] Training samples can be obtained through manual labeling or AI-assisted labeling.
[0046] The sample images here can be determined according to actual needs. For example, this application mainly targets semantic recognition in autonomous driving scenarios. The sample images can mainly be some images of roads and surrounding areas. These images can include objects such as vehicles, roads, green belts, pedestrians, trees, buildings, curbs, traffic equipment, zebra crossings, and lane lines.
[0047] In addition, the acquired images can be augmented to enrich the sample. For example, sample enrichment can be achieved through flipping, translating, cropping, rotating, and texture enhancement.
[0048] Here, the category can be the category corresponding to the aforementioned objects, and the category label can be the category code. For example, pedestrians are coded as 1, vehicles as coded as 2, and so on. The specific coding method and format can be determined according to actual needs.
[0049] The known categories here can be clearly defined or categories that require attention, such as vehicles, roads, green belts, pedestrians, trees, buildings, curbs, traffic equipment, zebra crossings, and lane markings. The other categories can be categories that do not require attention or are unclear. For example, if there are 10 categories in total, then 1-9 can be known categories, and 10 can be other categories.
[0050] Each pixel in the sample image corresponds to a category label. For each category in the image, there can be one or more regions, and all pixels within that region can correspond to that category. Each region can also have a category edge. These category edges can be defined by edge lines.
[0051] This contrastive feature can have an initial value, which can be determined empirically. The dimension of this contrastive feature can be the same as the dimension of the target semantic feature.
[0052] like Figure 2 As shown, the semantic segmentation model for this scene includes a feature extraction layer and a classification layer. The feature extraction layer includes contrast features.
[0053] The feature extraction layer may include an encoding layer and a decoding layer. For example, the feature extraction layer may be a feature pyramid model, and the classification layer may be a softmax layer.
[0054] For example, the semantic segmentation model for this scene could be the SegNet model, with contrastive features added to improve training performance. In the encoding layer, convolutional layers extract image features, followed by pooling layers that downsample the image (halving the width and height while maintaining the number of channels). These scale-invariant features are then passed to the next layer. Batch normalization (BN) layers perform batch normalization on the training images to accelerate learning. In the decoding layer, the scaled-down feature maps are upsampled, and then convolutional processing is applied to refine the geometry of objects in the image, restoring the features obtained in the encoding layer to the specific pixels of the original image. Finally, a softmax layer outputs the class distribution for each pixel.
[0055] S104, the scene semantic segmentation model extracts features from the target sample image to obtain target semantic features and determines the target category of the target semantic features.
[0056] After determining the training samples and the initial scene semantic segmentation model, the initial scene semantic segmentation model can be trained using the training samples.
[0057] The sample images from the training samples can be input into the initial scene semantic segmentation model. The feature extraction layer extracts the semantic encoding features corresponding to each pixel, and the classification layer classifies the category corresponding to each pixel.
[0058] In this process, multi-level semantic features can be obtained by extracting features from the target sample image using a scene semantic segmentation model. Then, the semantic features extracted from the feature image in the previous layer of the multi-level semantic features are determined as the target semantic features. These multi-level semantic features correspond to different dimensions; for example, features of different dimensions can be obtained by using convolutional kernels of different scales in a feature pyramid model.
[0059] Each pixel can correspond to a category label, an opposite feature, a target semantic feature, and a target category.
[0060] S106, Based on the target category, the label of the target sample image category, and the contrast features, the parameters of the initial scene semantic segmentation model are optimized according to the preset expectation to determine the trained scene semantic segmentation model.
[0061] The pre-set expectation here can be the capability that the semantic segmentation model of the expected scenario will acquire.
[0062] The expectation mainly includes: the category of each pixel is the same as its corresponding category label, and it has a certain ability to recognize unknown objects. That is, while considering the classification accuracy of known categories, the separation between unknown categories and known category space is also considered.
[0063] For example, the pre-set expectation here may include: the first distance between the target semantic feature and the opposite feature corresponding to the target category is greater than the second distance between the target semantic feature and other semantic features in the target category;
[0064] The target category is the same as the label category of the target sample image.
[0065] It can also include loss functions based on cross-entropy and the desired value of the classification margin, where the classification margin is the interval between the boundaries corresponding to the categories.
[0066] In some embodiments, the opposing features of the target category can also be optimized based on all semantic features of the target category. As an example, the third distance between all semantic features of the target category and the opposing features of the target category can be determined; the fourth distance between all semantic features of other categories and the opposing features of the target category can be determined; and the opposing features of the target category can be optimized based on a pre-defined optimization algorithm so that the sum of all third distances approaches infinity and the sum of all fourth distances approaches infinity.
[0067] In semantic segmentation, each pixel is ultimately projected into a target semantic feature (a vector) in a high-dimensional space. Target semantic features of pixels of the same category can cluster together in this high-dimensional space. For each category in semantic segmentation (except for the else category), an opposite feature p is defined in this high-dimensional space. This opposite feature can be used to describe the cluster center of all target semantic features of all else categories. Based on this, the opposite feature of each category should be as far away as possible from the target semantic feature cluster of that category. This relationship can then be expressed by formula (I):
[0068] Formula (1)
[0069] in, Indicates the first The opposite of class, Indicates the first The feature vector of each pixel in high-dimensional space.
[0070] To better describe unknown objects as being as far away as possible from known meaningful categories, all unknown objects can be categorized into the else class. Therefore, the defined opposition features should be as close as possible to the semantic feature vector cluster of the else class in high-dimensional space. This relationship can then be expressed by formula (II):
[0071] Formula (II)
[0072] in Indicates the first The opposite of class, Indicates the first The feature vectors of pixels in the else category in high-dimensional space.
[0073] Under the constraints of Formula (I) and Formula (II), the gradient descent method can be used to optimize the initial features of the opposing points to obtain the optimal features of the opposing points.
[0074] The embodiments of this application can train a model using the adversarial mechanism of the above formulas (I) and (II), and add a classification margin to the original softmax cross-entropy loss function. This can maximize the distance between each category in the high-dimensional space, increase the distance between classes, reduce the probability of misclassification between categories, and at the same time make unknown objects as much as possible to be classified into the else category without interfering with the segmentation of known meaningful category objects.
[0075] like Figure 3 As shown, this embodiment of the invention also provides a scene semantic segmentation model training device 200, the device comprising:
[0076] The first determining module 201 determines training samples and an initial scene semantic segmentation model. The training samples include sample images and labels corresponding to the categories of pixels in the sample images. The categories include multiple known categories and other categories. The scene semantic segmentation model includes contrast features corresponding to each known category and the contrast features correspond to other categories.
[0077] The second determining module 202 performs feature extraction on the target sample image by the scene semantic segmentation model to obtain target semantic features and determines the target category of the target semantic features;
[0078] Training module 203 optimizes the parameters of the initial scene semantic segmentation model according to a preset expectation based on the target category, the category label of the target sample image, and the opposite point features, and determines the trained scene semantic segmentation model.
[0079] The preset expectation includes that the first distance between the target semantic feature and the opposite feature corresponding to the target category is greater than the second distance between the target semantic feature and other semantic features in the target category; the target category is the same as the label category of the target sample image.
[0080] In some embodiments, an optimization module is also included to optimize the opposite features of the target category based on all semantic features of the target category.
[0081] In some embodiments, the optimization module is further configured to: determine the third distances between all semantic features of the target category and the opposing features of the target category; determine the fourth distances between all semantic features of other categories and the opposing features of the target category; and optimize the opposing features of the target category based on a pre-set optimization algorithm so that the sum of all third distances approaches infinity and the sum of all fourth distances approaches infinity.
[0082] In some embodiments, the second determining module 202 is further specifically used to: the scene semantic segmentation model extracts features from the target sample image to obtain multi-level semantic features; and determines the semantic features extracted from the feature image in the previous layer of the multi-level semantic features as the target semantic features.
[0083] In some embodiments, each pixel corresponds to a category label, an opposite feature, a target semantic feature, and a target category.
[0084] In some embodiments, the scene semantic segmentation model includes a feature extraction layer and a classification layer, wherein the feature extraction layer includes the opposing point features.
[0085] In some embodiments, the preset expectation further includes reaching a desired value based on the cross-entropy loss function and the classification interval, wherein the classification interval is the interval between the boundaries corresponding to the categories.
[0086] Figure 4 This is a schematic diagram of the hardware architecture of the electronic device 300 provided in an embodiment of the present invention. See also... Figure 4 As shown, the electronic device 300 includes a machine-readable storage medium 301 and a processor 302, and may also include a non-volatile storage medium 303, a communication interface 304, and a bus 305; wherein the machine-readable storage medium 301, the processor 302, the non-volatile storage medium 303, and the communication interface 304 communicate with each other through the bus 305. The processor 302 can execute the scene semantic segmentation model training method described in the above embodiments by reading and executing the machine-executable instructions for scene semantic segmentation model training in the machine-readable storage medium 301.
[0087] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.
[0088] Non-volatile media can be non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar non-volatile storage media, or combinations thereof.
[0089] It is understood that the specific operation methods of each functional module in this embodiment can be referred to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.
[0090] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program. When the computer program code is executed, it can implement the scene semantic segmentation model training method described in any of the above embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.
[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0092] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.
[0093] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0094] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.
Claims
1. A method for training a scene semantic segmentation model, characterized in that, include: Determine training samples and an initial scene semantic segmentation model. The training samples include sample images and labels corresponding to the categories of pixels in the sample images. The categories include multiple known categories and other categories. The other categories are unknown object categories that have not been seen during the training process. The scene semantic segmentation model includes contrast features corresponding to each known category. The contrast features are the semantic feature cluster centers of other categories in high-dimensional space. The scene semantic segmentation model extracts features from the target sample image to obtain target semantic features and determines the target category of the target semantic features; Based on the target category, the category label of the target sample image, and the opposite point features, the parameters of the initial scene semantic segmentation model are optimized according to the preset expectation to determine the trained scene semantic segmentation model. The preset expectation includes: a first distance between the target semantic feature and the opposite feature corresponding to the target category is greater than a second distance between the target semantic feature and other semantic features in the target category; the target category is the same as the label category of the target sample image; the opposite feature must satisfy that the sum of the third distances with all semantic features of the target category approaches infinity, and the sum of the fourth distances with all semantic features of other categories approaches infinity.
2. The method according to claim 1, characterized in that, Also includes: The opposing features of the target category are optimized based on all semantic features of the target category.
3. The method according to claim 2, characterized in that, The optimization of the opposite features of the target category based on all semantic features of the target category includes: Determine the third distance between all semantic features of the target category and the opposite feature of the target category; Determine the fourth distance between all semantic features of other categories and the opposite feature of the target category; The opposing features of the target category are optimized based on a pre-defined optimization algorithm so that the sum of all the third distances approaches infinity and the sum of all the fourth distances approaches infinity.
4. The method according to claim 1, characterized in that, The scene semantic segmentation model extracts features from the target sample image to obtain target semantic features, including: The scene semantic segmentation model extracts multi-level semantic features from the target sample image; The semantic features extracted from the feature image of the previous layer in the multi-level semantic features are determined as the target semantic features.
5. The method according to claim 1, characterized in that, Each pixel corresponds to a category label, an opposite feature, a target semantic feature, and a target category.
6. The method according to claim 1, characterized in that, The scene semantic segmentation model includes a feature extraction layer and a classification layer, wherein the feature extraction layer includes the opposing point features.
7. The method according to claim 1, characterized in that, The preset expectation also includes reaching the expected value based on the cross-entropy loss function and the classification interval, where the classification interval is the interval between the boundaries corresponding to the categories.
8. A scene semantic segmentation model training device, characterized in that, The device includes: The first determining module determines the training samples and the initial scene semantic segmentation model. The training samples include sample images and labels corresponding to the categories of the pixels in the sample images. The categories include multiple known categories and other categories. The other categories are unknown object categories that have not been seen during the training process. The scene semantic segmentation model includes the opposite point features corresponding to each known category. The opposite point features are the semantic feature cluster centers of other categories in high-dimensional space. The second determining module involves the scene semantic segmentation model extracting features from the target sample image to obtain target semantic features and determining the target category of the target semantic features. The training module optimizes the parameters of the initial scene semantic segmentation model according to a preset expectation based on the target category, the category label of the target sample image, and the opposite point features, and determines the trained scene semantic segmentation model. The preset expectation includes: a first distance between the target semantic feature and the opposite feature corresponding to the target category is greater than a second distance between the target semantic feature and other semantic features in the target category; the target category is the same as the label category of the target sample image; the opposite feature must satisfy that the sum of the third distances with all semantic features of the target category approaches infinity, and the sum of the fourth distances with all semantic features of other categories approaches infinity.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.
10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the steps of the method according to any one of claims 1 to 7.